Advanced Parameter Expansion
⚙️ Intermediate Topic 1 — Advanced Parameter Expansion
The beginner course introduced the essentials of parameter expansion — defaults, string length, substring removal, and basic substitution. This chapter goes much deeper. Bash's parameter expansion system is one of the most powerful features of the shell, capable of replacing entire awk and sed pipelines with a single expression. We cover every operator in the language, the bash 4.4+ transformation flags, indirect expansion, namerefs, and compound patterns that combine multiple expansions into concise, readable one-liners.
1 — Default, Assign, Error, and Alternate Operators
You know ${var:-default}. But there are four operators in this family, each with a colon and a no-colon variant — eight forms in total — and the distinction between them matters.
| Form | Meaning | Triggers when var is… |
|---|---|---|
${var:-val} | Use val if unset or empty | unset or "" |
${var-val} | Use val only if unset | unset only (empty string is fine) |
${var:=val} | Assign val if unset or empty, then expand | unset or "" |
${var=val} | Assign val only if unset, then expand | unset only |
${var:?msg} | Error and exit if unset or empty | unset or "" |
${var?msg} | Error and exit if unset | unset only |
${var:+val} | Use val if set and non-empty (opposite of :-) | when var has a value |
${var+val} | Use val if set at all (even if empty) | when var exists at all |
empty=""
unset gone
# :- vs - (empty string triggers :- but NOT -)
echo "${empty:-fallback}" → fallback (empty triggers it)
echo "${empty-fallback}" → (empty string returned — no trigger)
echo "${gone:-fallback}" → fallback
echo "${gone-fallback}" → fallback
# := assigns as a side-effect — useful for lazy initialisation
echo "${cache:=computed_value}" → computed_value (also sets $cache)
echo "$cache" → computed_value (now set)
# Cannot use := on positional parameters ($1, $2) or special vars
# :? aborts with a message — great for required arguments
db_host="${DB_HOST:?DB_HOST environment variable must be set}"
# If DB_HOST is unset: bash: DB_HOST: DB_HOST environment variable must be set
# :+ is the logical opposite of :- — "use this value IF the var IS set"
debug="1"
echo "${debug:+--verbose}" → --verbose (because debug is set)
echo "${unset_var:+--verbose}" → (nothing, because it's unset)
# Practical: conditionally add a flag to a command
curl "${verbose:+--silent}" https://example.com
# Passes --silent only when $verbose is empty/unset (backwards logic example)
# + without colon: even an empty string counts as "set"
empty=""
echo "${empty+was_set}" → was_set (empty var still "exists")
echo "${gone+was_set}" → (unset var — nothing)
:- := :? :+) treat an empty string the same as unset. The no-colon variants (- = ? +) only act when the variable is completely unset. In practice you almost always want the colon forms.2 — Substring Extraction and Prefix/Suffix Removal
Substring extraction — ${var:offset:length}
str="Hello, World!"
# ${var:offset} — from offset to end
echo "${str:7}" → World!
# ${var:offset:length}
echo "${str:7:5}" → World
echo "${str:0:5}" → Hello
# Negative offset counts from the END — note the space before the minus
# (space required to distinguish from :- default operator)
echo "${str: -6}" → orld!
echo "${str: -6:5}" → orld! wait — 5 chars from position -6
echo "${str: -6:4}" → orld
# Negative length means "exclude this many chars from the end"
echo "${str:7: -1}" → World (6 chars: "World!" minus last 1)
# Works on arrays too — returns a slice of elements
arr=( a b c d e f )
echo "${arr[@]:2:3}" → c d e
echo "${arr[@]: -2}" → e f
# Works on positional parameters
# Inside a function: ${@:2} = all args from the second onwards
all_but_first() { echo "${@:2}"; }
all_but_first one two three four
→ two three four
Pattern-based prefix and suffix removal — going deeper
path="/usr/local/lib/libmyapp.so.2.1.0"
# # removes shortest prefix matching the pattern
echo "${path#*/}" → usr/local/lib/libmyapp.so.2.1.0
# ## removes longest prefix matching the pattern
echo "${path##*/}" → libmyapp.so.2.1.0 (basename)
# % removes shortest suffix
echo "${path%/*}" → /usr/local/lib (dirname)
echo "${path%.*}" → /usr/local/lib/libmyapp.so.2.1
# %% removes longest suffix
echo "${path%%.*}" → /usr/local/lib/libmyapp
# Practical: strip all extensions from a filename
f="archive.tar.gz"
echo "${f%%.*}" → archive
# Replace extension: strip suffix then add new one
echo "${f%.gz}.bz2" → archive.tar.bz2
# Works in a loop — bulk rename files
for f in *.jpeg; do
mv "$f" "${f%.jpeg}.jpg"
done
# Combined: extract just the base filename without any extension
file="/var/log/myapp/error.log.1"
base="${file##*/}" # → error.log.1 (strip directory)
name="${base%%.*}" # → error (strip all extensions)
dir="${file%/*}" # → /var/log/myapp (dirname)
Search and replace — all four forms
str="the cat sat on the mat"
# / — replace FIRST occurrence
echo "${str/the/a}" → a cat sat on the mat
# // — replace ALL occurrences
echo "${str//the/a}" → a cat sat on a mat
# /# — replace only if pattern matches at the START
echo "${str/#the/a}" → a cat sat on the mat
echo "${str/#cat/a}" → the cat sat on the mat (no match — not at start)
# /% — replace only if pattern matches at the END
echo "${str/%mat/rug}" → the cat sat on the rug
# Replace with empty string = deletion
echo "${str// /}" → thecatsatonthemat (remove all spaces)
echo "${str//[aeiou]/}" → th ct st n th mt (remove vowels)
# Patterns support globs
csv="one, two , three "
echo "${csv// /}" → one,two,three (remove all spaces)
# Works on each element of an array (expands to modified array)
names=( "Alice Smith" "Bob Jones" "Carol White" )
echo "${names[@]// /_}" → Alice_Smith Bob_Jones Carol_White
3 — Case Conversion Operators
Bash 4.0 added four case-conversion operators. They're useful for normalising user input, building slugs, and formatting output without spawning tr or awk.
str="Hello World"
# ^ — uppercase first character only
lower="hello world"
echo "${lower^}" → Hello world
# ^^ — uppercase ALL characters
echo "${lower^^}" → HELLO WORLD
# , — lowercase first character only
echo "${str,}" → hello World
# ,, — lowercase ALL characters
echo "${str,,}" → hello world
# ~ — toggle case of first character
echo "${str~}" → hello World
# ~~ — toggle case of ALL characters
echo "${str~~}" → hELLO wORLD
# All operators accept a glob pattern — only matching chars are converted
echo "${lower^^[aeiou]}" → hEllO wOrld (only vowels uppercased)
echo "${str,,[A-Z]}" → hello world (only uppercase letters lowercased)
# Practical: normalise input for case-insensitive comparison
read -r -p "Continue? [y/n]: " ans
if [[ "${ans,,}" == "y" || "${ans,,}" == "yes" ]]; then
echo "Proceeding..."
fi
# Build a slug from a title
title="My Blog Post Title"
slug="${title,,}" # → "my blog post title"
slug="${slug// /-}" # → "my-blog-post-title"
/bin/bash may still be version 3.2 (due to GPL licensing). If portability to macOS system bash matters, use tr instead, or ensure a newer bash is installed via Homebrew.4 — Indirect Expansion: ${!var}
The ! prefix makes bash expand the value of a variable as the name of another variable to expand. This allows dynamic variable lookup and is the classic way to implement dispatch tables without associative arrays.
fruit="apple"
apple="green"
# Normal expansion
echo "$fruit" → apple
# Indirect: expand $fruit to get the name "apple", then expand $apple
echo "${!fruit}" → green
# Indirect expansion with a variable holding a variable name
varname="PATH"
echo "${!varname}" → /usr/local/bin:/usr/bin:/bin:...
# Practical: environment-specific config lookup
env="prod"
prod_host="db.example.com"
staging_host="db-staging.example.com"
dev_host="localhost"
host_var="${env}_host" # → "prod_host"
echo "${!host_var}" → db.example.com
# Dispatch table: map command names to function names
cmd_start="do_start"
cmd_stop="do_stop"
cmd_status="do_status"
do_start() { echo "Starting..."; }
do_stop() { echo "Stopping..."; }
do_status() { echo "Running."; }
action="start"
fn_var="cmd_${action}" # → "cmd_start"
"${!fn_var}" # calls do_start
Starting...
Listing variables by prefix — ${!prefix*} and ${!prefix@}
DB_HOST="localhost"
DB_PORT="5432"
DB_NAME="myapp"
APP_ENV="production"
# ${!prefix*} expands to all variable NAMES starting with prefix
echo "${!DB_*}" → DB_HOST DB_NAME DB_PORT
# Iterate over all DB_ variables and print their names + values
for varname in "${!DB_@}"; do
printf "%-15s = %s\n" "$varname" "${!varname}"
done
DB_HOST = localhost
DB_NAME = myapp
DB_PORT = 5432
# Useful for printing a config dump at script startup
dump_config() {
echo "Configuration:"
local v
for v in "${!APP_@}" "${!DB_@}"; do
printf " %s=%s\n" "$v" "${!v}"
done
}
# ${!prefix*} vs ${!prefix@}
# When unquoted: identical
# When quoted: * joins with IFS; @ produces separate words (like $* vs $@)
5 — Namerefs: declare -n
A nameref is a variable that acts as an alias for another variable — named at runtime. This is the clean, modern solution to passing arrays in and out of functions, and removes the need for the fragile ${!varname} indirect expansion pattern in most cases.
# declare -n creates a nameref — the variable IS the other variable
original="hello"
declare -n ref="original"
echo "$ref" → hello
ref="world" # modifying ref modifies original
echo "$original" → world
# A nameref to an array — you get the full array
colours=( red green blue )
declare -n arr_ref="colours"
echo "${arr_ref[1]}" → green
arr_ref+=( yellow )
echo "${colours[@]}" → red green blue yellow
# Pass the name of the caller's variable; the function writes to it
get_stats() {
local -n _result="$1" # _result IS the caller's variable
local total=0 count=0 min max
shift # remaining args are the numbers
min=$1; max=$1
for n in "$@"; do
(( total += n, count++ ))
(( n < min )) && min=$n
(( n > max )) && max=$n
done
# Write into an associative array via the nameref
_result[count]=$count
_result[total]=$total
_result[min]=$min
_result[max]=$max
_result[mean]=$(echo "scale=2; $total/$count" | bc)
}
# Caller declares the output map first, then passes its name
declare -A stats
get_stats stats 5 10 3 8 15 2
printf "Count: %d Total: %d Min: %d Max: %d Mean: %s\n" \
"${stats[count]}" "${stats[total]}" \
"${stats[min]}" "${stats[max]}" "${stats[mean]}"
Count: 6 Total: 43 Min: 2 Max: 15 Mean: 7.16
# Returning an indexed array from a function
split_csv() {
local -n _out="$1"
IFS=',' read -r -a _out <<< "$2"
}
split_csv parts "alpha,beta,gamma,delta"
echo "${parts[2]}" → gamma
_result) inside functions. If the caller passes in a variable name that happens to match your local nameref name, bash will produce a circular reference error. The underscore prefix makes collisions essentially impossible in practice.6 — Transformation Operators: ${var@operator}
Bash 4.4 added a family of transformation operators using the @ suffix. These are less well-known but extremely useful for safe quoting, debugging, and code generation.
echo $BASH_VERSION to check.| Operator | What it produces | Example |
|---|---|---|
${var@Q} | Quoted form — safe to re-use in shell input | "hello world" → 'hello world' |
${var@E} | Expands escape sequences like $'...' | "tab:\t" → tab: |
${var@P} | Expanded as a prompt string (like PS1) | "\u@\h" → alice@myhost |
${var@A} | Assignment form — declare statement to recreate the variable | declare -- myvar='hello' |
${var@a} | Attribute flags of the variable | r for readonly, A for assoc array, etc. |
${var@U} | Uppercase (alias for ${var^^}) | bash 5.1+ |
${var@u} | First char uppercase (alias for ${var^}) | bash 5.1+ |
${var@L} | Lowercase (alias for ${var,,}) | bash 5.1+ |
# @Q wraps the value in single quotes (or $'...' for special chars)
# making it safe to embed in eval, log output, or generated scripts
name="Alice O'Brien"
echo "${name@Q}" → 'Alice O'\''Brien' (properly escaped)
path="/home/alice/my documents"
echo "${path@Q}" → '/home/alice/my documents'
# Log a command in a way that can be copy-pasted and re-run
log_command() {
local quoted=()
local arg
for arg in "$@"; do
quoted+=( "${arg@Q}" )
done
echo "[CMD] ${quoted[*]}" >&2
}
log_command cp "file with spaces.txt" "/dest/path"
[CMD] cp 'file with spaces.txt' '/dest/path'
# @A — get the declaration form of a variable
myvar="hello world"
echo "${myvar@A}" → declare -- myvar='hello world'
declare -A config=( [host]=localhost [port]=5432 )
echo "${config@A}" → declare -A config=([host]="localhost" [port]="5432" )
# This is exactly what declare -p produces — useful for serialising to a file
declare -r READONLY_VAR="fixed"
declare -i INT_VAR=42
declare -a INDEXED=( a b c )
declare -A ASSOC=( [k]=v )
declare -x EXPORTED="val"
declare -l LOWER_VAR="INPUT"
echo "${READONLY_VAR@a}" → r
echo "${INT_VAR@a}" → i
echo "${INDEXED@a}" → a
echo "${ASSOC@a}" → A
echo "${EXPORTED@a}" → x
echo "${LOWER_VAR@a}" → l (auto-lowercase attribute)
# Use @a in a guard to check a variable is the right type
if [[ "${myvar@a}" == *"A"* ]]; then
echo "myvar is an associative array"
fi
7 — Compound and Nested Expansions
Parameter expansions can be nested and combined. The patterns below replace entire pipelines with single in-shell expressions — no subshell, no fork, no external process.
# ── Combining operations ─────────────────────────────────────
# Trim leading whitespace (no sed needed)
str=" hello world "
trimmed="${str#"${str%%[! ]*}"}" # strip leading spaces
echo "'$trimmed'" → 'hello world '
# Trim trailing whitespace
trimmed="${trimmed%"${trimmed##*[! ]}"}"
echo "'$trimmed'" → 'hello world'
# ── Default with transformation ──────────────────────────────
# Uppercase the value, or use a default if unset
mode="production"
echo "${mode^^:-UNKNOWN}" → PRODUCTION
# The above doesn't nest — you need two steps for that:
mode_upper="${mode:-unknown}" # apply default first
echo "${mode_upper^^}" → PRODUCTION
# ── Dynamic variable names via indirection ────────────────────
# Build variable names on the fly and look them up
region="eu"
eu_endpoint="https://eu.api.example.com"
us_endpoint="https://us.api.example.com"
endpoint_var="${region}_endpoint"
echo "${!endpoint_var}" → https://eu.api.example.com
# ── Length of a transformed value ────────────────────────────
word="Hello"
echo "${#word}" → 5
# You can't do ${#word^^} directly — assign to intermediate var
upper="${word^^}"
echo "${#upper}" → 5 (same length, different case)
# ── Array expansion with per-element transformation ──────────
tags=( bash linux scripting )
# Uppercase every element
echo "${tags[@]^^}" → BASH LINUX SCRIPTING
# Replace hyphens with underscores in every element
keys=( "my-key" "another-key" "third-key" )
echo "${keys[@]//-/_}" → my_key another_key third_key
8 — Quick Reference
Default / assign / error / alternate
| Expansion | Expands to | Side effect |
|---|---|---|
${v:-val} | val if v unset or empty, else v | none |
${v-val} | val if v unset, else v | none |
${v:=val} | val if v unset or empty, else v | assigns v=val |
${v=val} | val if v unset, else v | assigns v=val |
${v:?msg} | v (if set and non-empty) | error+exit if unset/empty |
${v?msg} | v (if set) | error+exit if unset |
${v:+val} | val if v set and non-empty, else empty | none |
${v+val} | val if v set at all, else empty | none |
String / substring operations
| Expansion | Result |
|---|---|
${#v} | Length of v in characters |
${v:n} | Substring from position n to end |
${v:n:len} | Substring: len chars from position n |
${v: -n} | Last n characters (space before minus) |
${v#pat} | Remove shortest prefix matching pat |
${v##pat} | Remove longest prefix matching pat |
${v%pat} | Remove shortest suffix matching pat |
${v%%pat} | Remove longest suffix matching pat |
${v/pat/rep} | Replace first match of pat with rep |
${v//pat/rep} | Replace all matches of pat with rep |
${v/#pat/rep} | Replace pat only at start |
${v/%pat/rep} | Replace pat only at end |
${v^} | Uppercase first char bash 4.0+ |
${v^^} | Uppercase all chars bash 4.0+ |
${v,} | Lowercase first char bash 4.0+ |
${v,,} | Lowercase all chars bash 4.0+ |
Indirection and transformation
| Expansion | Result |
|---|---|
${!v} | Value of the variable whose name is stored in v |
${!prefix@} | Names of all variables starting with prefix |
${v@Q} | Shell-quoted form of v bash 4.4+ |
${v@A} | declare statement to recreate v bash 4.4+ |
${v@a} | Attribute flags of v (r=readonly, i=integer, a=array…) bash 4.4+ |
${v@E} | Expand escape sequences in v bash 4.4+ |
declare -n ref=name | Make ref an alias for the variable named name bash 4.3+ |
✏️ Exercises
Each exercise is designed to be solved purely with parameter expansion — resist the urge to reach for sed, awk, or python. The goal is to develop fluency with the shell's built-in string machinery.
path_info() that accepts a full file path as its only argument and — using only parameter expansion, no external commands — prints four lines: the directory, the filename (with extension), the base name (without any extension), and the extension (without the dot). Test it with /var/log/nginx/access.log.2.${path%/*}, filename = ${path##*/}, extension = ${filename##*.}, base = ${filename%%.*}. Edge case: a file with no extension.#!/usr/bin/env bash
set -euo pipefail
path_info() {
local path="${1:?path_info: argument required}"
local dir filename base ext
dir="${path%/*}"
filename="${path##*/}"
# Only split extension if there is a dot in the filename
if [[ "$filename" == *"."* ]]; then
ext="${filename##*.}"
base="${filename%%.*}"
else
ext=""
base="$filename"
fi
printf "Directory : %s\n" "$dir"
printf "Filename : %s\n" "$filename"
printf "Base name : %s\n" "$base"
printf "Extension : %s\n" "${ext:-(none)}"
}
path_info "/var/log/nginx/access.log.2"
Directory : /var/log/nginx
Filename : access.log.2
Base name : access
Extension : log.2
path_info "/usr/bin/bash"
Directory : /usr/bin
Filename : bash
Base name : bash
Extension : (none)
return_multiple() that calculates the quotient and remainder of two integers (passed as arguments), and returns both values to the caller using a nameref — so the caller can use them as separate variables. Also write a second function swap() that uses two namerefs to swap the values of two caller-side variables in place, without using a temporary variable visible to the caller.return_multiple, accept two nameref names as the first two arguments. For swap, use local -n _a="$1" _b="$2" and a local temp variable for the swap itself.#!/usr/bin/env bash
set -euo pipefail
divmod() {
# Usage: divmod QUOT_VAR REM_VAR DIVIDEND DIVISOR
local -n _quot="$1"
local -n _rem="$2"
local _a="$3" _b="$4"
[[ "$_b" -ne 0 ]] || { echo "divmod: division by zero" >&2; return 1; }
_quot=$(( _a / _b ))
_rem=$(( _a % _b ))
}
swap() {
# Usage: swap VAR1 VAR2 — swaps values in place
local -n _x="$1"
local -n _y="$2"
local _tmp="$_x"
_x="$_y"
_y="$_tmp"
}
# Test divmod
declare -i q r
divmod q r 17 5
echo "17 ÷ 5 = $q remainder $r"
17 ÷ 5 = 3 remainder 2
# Test swap
x="hello"; y="world"
echo "Before: x=$x y=$y"
swap x y
echo "After: x=$x y=$y"
Before: x=hello y=world
After: x=world y=hello
config_check.sh that uses ${var:?} to validate a set of required environment variables (APP_ENV, DB_HOST, DB_PORT, SECRET_KEY). For optional variables (LOG_LEVEL, TIMEOUT), provide sensible defaults using ${var:-default}. Then print a startup summary showing each variable's name, its value, and its source ("required", "default", or "env"). Use ${!prefix@} to discover and print all APP_ and DB_ variables automatically.if [[ -n "${LOG_LEVEL+x}" ]]; then src="env"; else src="default"; fi. Then apply the default. For the prefix listing, loop over "${!APP_@}" and "${!DB_@}".#!/usr/bin/env bash
# config_check.sh
set -uo pipefail # no -e so :? error message is visible before exit
# ── Required variables — script exits here if any are unset/empty ──
APP_ENV="${APP_ENV:?APP_ENV is required (e.g. production, staging, dev)}"
DB_HOST="${DB_HOST:?DB_HOST is required}"
DB_PORT="${DB_PORT:?DB_PORT is required}"
SECRET_KEY="${SECRET_KEY:?SECRET_KEY is required}"
# ── Optional variables — track source before applying default ──
resolve_optional() {
local -n _var="$1"
local _default="$2"
if [[ -n "${_var+x}" && -n "${_var}" ]]; then
echo "env"
else
_var="$_default"
echo "default"
fi
}
ll_src=$(resolve_optional LOG_LEVEL "INFO")
to_src=$(resolve_optional TIMEOUT "30")
# ── Print startup summary ─────────────────────────────────────
echo
printf "\033[1mConfiguration Summary\033[0m\n"
printf '%.0s─' {1..50}; echo
printf "%-18s %-25s %s\n" "Variable" "Value" "Source"
printf '%.0s─' {1..50}; echo
# Required
for v in APP_ENV DB_HOST DB_PORT; do
# Mask SECRET_KEY
display="${!v}"
printf "%-18s %-25s \033[32mrequired\033[0m\n" "$v" "$display"
done
# Show masked secret
masked="${SECRET_KEY:0:4}****${SECRET_KEY: -2}"
printf "%-18s %-25s \033[32mrequired\033[0m\n" "SECRET_KEY" "$masked"
# Optional
printf "%-18s %-25s \033[33m%s\033[0m\n" "LOG_LEVEL" "$LOG_LEVEL" "$ll_src"
printf "%-18s %-25s \033[33m%s\033[0m\n" "TIMEOUT" "$TIMEOUT" "$to_src"
printf '%.0s─' {1..50}; echo
echo
echo "All APP_ and DB_ variables in scope:"
for v in "${!APP_@}" "${!DB_@}"; do
printf " %s=%s\n" "$v" "${!v}"
done
safe_log_args() that takes any number of arguments (as a command and its arguments would be passed to it), and logs them in a copy-pasteable form using ${arg@Q} to handle values containing spaces, quotes, or special characters. Then write a wrapper function run() that calls safe_log_args before executing its arguments as a command. Demonstrate it wrapping cp and mkdir with tricky paths.safe_log_args(), loop over "$@" and collect "${arg@Q}" into an array, then print the array joined with spaces. In run(), call safe_log_args "$@" first, then "$@" to execute. Test with paths containing spaces.#!/usr/bin/env bash
set -euo pipefail
safe_log_args() {
local quoted=()
local arg
for arg in "$@"; do
quoted+=( "${arg@Q}" )
done
# Print as a reproducible shell command
printf '\033[36m[RUN]\033[0m %s\n' "${quoted[*]}" >&2
}
run() {
safe_log_args "$@"
"$@"
}
# Demonstration
TMPDIR=$(mktemp -d)
trap 'rm -rf "$TMPDIR"' EXIT
# These commands have spaces and special chars in paths
run mkdir -p "${TMPDIR}/my project/src files"
run touch "${TMPDIR}/my project/src files/main script.sh"
run cp "${TMPDIR}/my project/src files/main script.sh" \
"${TMPDIR}/backup of main script.sh"
[RUN] mkdir -p '/tmp/tmp.abc123/my project/src files'
[RUN] touch '/tmp/tmp.abc123/my project/src files/main script.sh'
[RUN] cp '/tmp/tmp.abc123/my project/src files/main script.sh' '/tmp/tmp.abc123/backup of main script.sh'
# The log output is safe to copy-paste and re-run exactly