Advanced Parameter Expansion

⚙️ Intermediate Topic 1 — Advanced Parameter Expansion

The beginner course introduced the essentials of parameter expansion — defaults, string length, substring removal, and basic substitution. This chapter goes much deeper. Bash's parameter expansion system is one of the most powerful features of the shell, capable of replacing entire awk and sed pipelines with a single expression. We cover every operator in the language, the bash 4.4+ transformation flags, indirect expansion, namerefs, and compound patterns that combine multiple expansions into concise, readable one-liners.

1 — Default, Assign, Error, and Alternate Operators

You know ${var:-default}. But there are four operators in this family, each with a colon and a no-colon variant — eight forms in total — and the distinction between them matters.

FormMeaningTriggers when var is…
${var:-val}Use val if unset or emptyunset or ""
${var-val}Use val only if unsetunset only (empty string is fine)
${var:=val}Assign val if unset or empty, then expandunset or ""
${var=val}Assign val only if unset, then expandunset only
${var:?msg}Error and exit if unset or emptyunset or ""
${var?msg}Error and exit if unsetunset only
${var:+val}Use val if set and non-empty (opposite of :-)when var has a value
${var+val}Use val if set at all (even if empty)when var exists at all
🐧 The colon makes all the difference
empty="" unset gone # :- vs - (empty string triggers :- but NOT -) echo "${empty:-fallback}" → fallback (empty triggers it) echo "${empty-fallback}" → (empty string returned — no trigger) echo "${gone:-fallback}" → fallback echo "${gone-fallback}" → fallback # := assigns as a side-effect — useful for lazy initialisation echo "${cache:=computed_value}" → computed_value (also sets $cache) echo "$cache" → computed_value (now set) # Cannot use := on positional parameters ($1, $2) or special vars # :? aborts with a message — great for required arguments db_host="${DB_HOST:?DB_HOST environment variable must be set}" # If DB_HOST is unset: bash: DB_HOST: DB_HOST environment variable must be set # :+ is the logical opposite of :- — "use this value IF the var IS set" debug="1" echo "${debug:+--verbose}" → --verbose (because debug is set) echo "${unset_var:+--verbose}" → (nothing, because it's unset) # Practical: conditionally add a flag to a command curl "${verbose:+--silent}" https://example.com # Passes --silent only when $verbose is empty/unset (backwards logic example) # + without colon: even an empty string counts as "set" empty="" echo "${empty+was_set}" → was_set (empty var still "exists") echo "${gone+was_set}" → (unset var — nothing)
The colon variants (:- := :? :+) treat an empty string the same as unset. The no-colon variants (- = ? +) only act when the variable is completely unset. In practice you almost always want the colon forms.

2 — Substring Extraction and Prefix/Suffix Removal

Substring extraction — ${var:offset:length}

🐧 Slicing strings with offset and length
str="Hello, World!" # ${var:offset} — from offset to end echo "${str:7}" → World! # ${var:offset:length} echo "${str:7:5}" → World echo "${str:0:5}" → Hello # Negative offset counts from the END — note the space before the minus # (space required to distinguish from :- default operator) echo "${str: -6}" → orld! echo "${str: -6:5}" → orld! wait — 5 chars from position -6 echo "${str: -6:4}" → orld # Negative length means "exclude this many chars from the end" echo "${str:7: -1}" → World (6 chars: "World!" minus last 1) # Works on arrays too — returns a slice of elements arr=( a b c d e f ) echo "${arr[@]:2:3}" → c d e echo "${arr[@]: -2}" → e f # Works on positional parameters # Inside a function: ${@:2} = all args from the second onwards all_but_first() { echo "${@:2}"; } all_but_first one two three four → two three four

Pattern-based prefix and suffix removal — going deeper

🐧 # ## % %% — with full glob patterns
path="/usr/local/lib/libmyapp.so.2.1.0" # # removes shortest prefix matching the pattern echo "${path#*/}" → usr/local/lib/libmyapp.so.2.1.0 # ## removes longest prefix matching the pattern echo "${path##*/}" → libmyapp.so.2.1.0 (basename) # % removes shortest suffix echo "${path%/*}" → /usr/local/lib (dirname) echo "${path%.*}" → /usr/local/lib/libmyapp.so.2.1 # %% removes longest suffix echo "${path%%.*}" → /usr/local/lib/libmyapp # Practical: strip all extensions from a filename f="archive.tar.gz" echo "${f%%.*}" → archive # Replace extension: strip suffix then add new one echo "${f%.gz}.bz2" → archive.tar.bz2 # Works in a loop — bulk rename files for f in *.jpeg; do mv "$f" "${f%.jpeg}.jpg" done # Combined: extract just the base filename without any extension file="/var/log/myapp/error.log.1" base="${file##*/}" # → error.log.1 (strip directory) name="${base%%.*}" # → error (strip all extensions) dir="${file%/*}" # → /var/log/myapp (dirname)

Search and replace — all four forms

🐧 / // /# /% — replace first, all, prefix, suffix
str="the cat sat on the mat" # / — replace FIRST occurrence echo "${str/the/a}" → a cat sat on the mat # // — replace ALL occurrences echo "${str//the/a}" → a cat sat on a mat # /# — replace only if pattern matches at the START echo "${str/#the/a}" → a cat sat on the mat echo "${str/#cat/a}" → the cat sat on the mat (no match — not at start) # /% — replace only if pattern matches at the END echo "${str/%mat/rug}" → the cat sat on the rug # Replace with empty string = deletion echo "${str// /}" → thecatsatonthemat (remove all spaces) echo "${str//[aeiou]/}" → th ct st n th mt (remove vowels) # Patterns support globs csv="one, two , three " echo "${csv// /}" → one,two,three (remove all spaces) # Works on each element of an array (expands to modified array) names=( "Alice Smith" "Bob Jones" "Carol White" ) echo "${names[@]// /_}" → Alice_Smith Bob_Jones Carol_White

3 — Case Conversion Operators

Bash 4.0 added four case-conversion operators. They're useful for normalising user input, building slugs, and formatting output without spawning tr or awk.

🐧 ^ ^^ , ,, ~ ~~ — case conversion
str="Hello World" # ^ — uppercase first character only lower="hello world" echo "${lower^}" → Hello world # ^^ — uppercase ALL characters echo "${lower^^}" → HELLO WORLD # , — lowercase first character only echo "${str,}" → hello World # ,, — lowercase ALL characters echo "${str,,}" → hello world # ~ — toggle case of first character echo "${str~}" → hello World # ~~ — toggle case of ALL characters echo "${str~~}" → hELLO wORLD # All operators accept a glob pattern — only matching chars are converted echo "${lower^^[aeiou]}" → hEllO wOrld (only vowels uppercased) echo "${str,,[A-Z]}" → hello world (only uppercase letters lowercased) # Practical: normalise input for case-insensitive comparison read -r -p "Continue? [y/n]: " ans if [[ "${ans,,}" == "y" || "${ans,,}" == "yes" ]]; then echo "Proceeding..." fi # Build a slug from a title title="My Blog Post Title" slug="${title,,}" # → "my blog post title" slug="${slug// /-}" # → "my-blog-post-title"
The case operators require bash 4.0+. On macOS, the system /bin/bash may still be version 3.2 (due to GPL licensing). If portability to macOS system bash matters, use tr instead, or ensure a newer bash is installed via Homebrew.

4 — Indirect Expansion: ${!var}

The ! prefix makes bash expand the value of a variable as the name of another variable to expand. This allows dynamic variable lookup and is the classic way to implement dispatch tables without associative arrays.

🐧 Variable indirection — ${!varname}
fruit="apple" apple="green" # Normal expansion echo "$fruit" → apple # Indirect: expand $fruit to get the name "apple", then expand $apple echo "${!fruit}" → green # Indirect expansion with a variable holding a variable name varname="PATH" echo "${!varname}" → /usr/local/bin:/usr/bin:/bin:... # Practical: environment-specific config lookup env="prod" prod_host="db.example.com" staging_host="db-staging.example.com" dev_host="localhost" host_var="${env}_host" # → "prod_host" echo "${!host_var}" → db.example.com # Dispatch table: map command names to function names cmd_start="do_start" cmd_stop="do_stop" cmd_status="do_status" do_start() { echo "Starting..."; } do_stop() { echo "Stopping..."; } do_status() { echo "Running."; } action="start" fn_var="cmd_${action}" # → "cmd_start" "${!fn_var}" # calls do_start Starting...

Listing variables by prefix — ${!prefix*} and ${!prefix@}

🐧 Enumerate all variables matching a prefix
DB_HOST="localhost" DB_PORT="5432" DB_NAME="myapp" APP_ENV="production" # ${!prefix*} expands to all variable NAMES starting with prefix echo "${!DB_*}" → DB_HOST DB_NAME DB_PORT # Iterate over all DB_ variables and print their names + values for varname in "${!DB_@}"; do printf "%-15s = %s\n" "$varname" "${!varname}" done DB_HOST = localhost DB_NAME = myapp DB_PORT = 5432 # Useful for printing a config dump at script startup dump_config() { echo "Configuration:" local v for v in "${!APP_@}" "${!DB_@}"; do printf " %s=%s\n" "$v" "${!v}" done } # ${!prefix*} vs ${!prefix@} # When unquoted: identical # When quoted: * joins with IFS; @ produces separate words (like $* vs $@)

5 — Namerefs: declare -n

A nameref is a variable that acts as an alias for another variable — named at runtime. This is the clean, modern solution to passing arrays in and out of functions, and removes the need for the fragile ${!varname} indirect expansion pattern in most cases.

Requires bash 4.3+. Namerefs were introduced in bash 4.3 (2014). They are available on all modern Linux distributions. macOS system bash (3.2) does not support them.
🐧 Nameref basics
# declare -n creates a nameref — the variable IS the other variable original="hello" declare -n ref="original" echo "$ref" → hello ref="world" # modifying ref modifies original echo "$original" → world # A nameref to an array — you get the full array colours=( red green blue ) declare -n arr_ref="colours" echo "${arr_ref[1]}" → green arr_ref+=( yellow ) echo "${colours[@]}" → red green blue yellow
🐧 Namerefs in functions — the right way to return arrays
# Pass the name of the caller's variable; the function writes to it get_stats() { local -n _result="$1" # _result IS the caller's variable local total=0 count=0 min max shift # remaining args are the numbers min=$1; max=$1 for n in "$@"; do (( total += n, count++ )) (( n < min )) && min=$n (( n > max )) && max=$n done # Write into an associative array via the nameref _result[count]=$count _result[total]=$total _result[min]=$min _result[max]=$max _result[mean]=$(echo "scale=2; $total/$count" | bc) } # Caller declares the output map first, then passes its name declare -A stats get_stats stats 5 10 3 8 15 2 printf "Count: %d Total: %d Min: %d Max: %d Mean: %s\n" \ "${stats[count]}" "${stats[total]}" \ "${stats[min]}" "${stats[max]}" "${stats[mean]}" Count: 6 Total: 43 Min: 2 Max: 15 Mean: 7.16 # Returning an indexed array from a function split_csv() { local -n _out="$1" IFS=',' read -r -a _out <<< "$2" } split_csv parts "alpha,beta,gamma,delta" echo "${parts[2]}" → gamma
Always name your nameref variables with a leading underscore (e.g. _result) inside functions. If the caller passes in a variable name that happens to match your local nameref name, bash will produce a circular reference error. The underscore prefix makes collisions essentially impossible in practice.

6 — Transformation Operators: ${var@operator}

Bash 4.4 added a family of transformation operators using the @ suffix. These are less well-known but extremely useful for safe quoting, debugging, and code generation.

Requires bash 4.4+ (released 2016). Available on Ubuntu 18.04+, Debian 9+, and any modern distro. Run echo $BASH_VERSION to check.
OperatorWhat it producesExample
${var@Q}Quoted form — safe to re-use in shell input"hello world"'hello world'
${var@E}Expands escape sequences like $'...'"tab:\t"tab:
${var@P}Expanded as a prompt string (like PS1)"\u@\h"alice@myhost
${var@A}Assignment form — declare statement to recreate the variabledeclare -- myvar='hello'
${var@a}Attribute flags of the variabler for readonly, A for assoc array, etc.
${var@U}Uppercase (alias for ${var^^})bash 5.1+
${var@u}First char uppercase (alias for ${var^})bash 5.1+
${var@L}Lowercase (alias for ${var,,})bash 5.1+
🐧 @Q — safe quoting, the most practically useful operator
# @Q wraps the value in single quotes (or $'...' for special chars) # making it safe to embed in eval, log output, or generated scripts name="Alice O'Brien" echo "${name@Q}" → 'Alice O'\''Brien' (properly escaped) path="/home/alice/my documents" echo "${path@Q}" → '/home/alice/my documents' # Log a command in a way that can be copy-pasted and re-run log_command() { local quoted=() local arg for arg in "$@"; do quoted+=( "${arg@Q}" ) done echo "[CMD] ${quoted[*]}" >&2 } log_command cp "file with spaces.txt" "/dest/path" [CMD] cp 'file with spaces.txt' '/dest/path' # @A — get the declaration form of a variable myvar="hello world" echo "${myvar@A}" → declare -- myvar='hello world' declare -A config=( [host]=localhost [port]=5432 ) echo "${config@A}" → declare -A config=([host]="localhost" [port]="5432" ) # This is exactly what declare -p produces — useful for serialising to a file
🐧 @a — inspecting variable attributes
declare -r READONLY_VAR="fixed" declare -i INT_VAR=42 declare -a INDEXED=( a b c ) declare -A ASSOC=( [k]=v ) declare -x EXPORTED="val" declare -l LOWER_VAR="INPUT" echo "${READONLY_VAR@a}" → r echo "${INT_VAR@a}" → i echo "${INDEXED@a}" → a echo "${ASSOC@a}" → A echo "${EXPORTED@a}" → x echo "${LOWER_VAR@a}" → l (auto-lowercase attribute) # Use @a in a guard to check a variable is the right type if [[ "${myvar@a}" == *"A"* ]]; then echo "myvar is an associative array" fi

7 — Compound and Nested Expansions

Parameter expansions can be nested and combined. The patterns below replace entire pipelines with single in-shell expressions — no subshell, no fork, no external process.

🐧 Chaining and nesting expansions
# ── Combining operations ───────────────────────────────────── # Trim leading whitespace (no sed needed) str=" hello world " trimmed="${str#"${str%%[! ]*}"}" # strip leading spaces echo "'$trimmed'" → 'hello world ' # Trim trailing whitespace trimmed="${trimmed%"${trimmed##*[! ]}"}" echo "'$trimmed'" → 'hello world' # ── Default with transformation ────────────────────────────── # Uppercase the value, or use a default if unset mode="production" echo "${mode^^:-UNKNOWN}" → PRODUCTION # The above doesn't nest — you need two steps for that: mode_upper="${mode:-unknown}" # apply default first echo "${mode_upper^^}" → PRODUCTION # ── Dynamic variable names via indirection ──────────────────── # Build variable names on the fly and look them up region="eu" eu_endpoint="https://eu.api.example.com" us_endpoint="https://us.api.example.com" endpoint_var="${region}_endpoint" echo "${!endpoint_var}" → https://eu.api.example.com # ── Length of a transformed value ──────────────────────────── word="Hello" echo "${#word}" → 5 # You can't do ${#word^^} directly — assign to intermediate var upper="${word^^}" echo "${#upper}" → 5 (same length, different case) # ── Array expansion with per-element transformation ────────── tags=( bash linux scripting ) # Uppercase every element echo "${tags[@]^^}" → BASH LINUX SCRIPTING # Replace hyphens with underscores in every element keys=( "my-key" "another-key" "third-key" ) echo "${keys[@]//-/_}" → my_key another_key third_key

8 — Quick Reference

Default / assign / error / alternate

ExpansionExpands toSide effect
${v:-val}val if v unset or empty, else vnone
${v-val}val if v unset, else vnone
${v:=val}val if v unset or empty, else vassigns v=val
${v=val}val if v unset, else vassigns v=val
${v:?msg}v (if set and non-empty)error+exit if unset/empty
${v?msg}v (if set)error+exit if unset
${v:+val}val if v set and non-empty, else emptynone
${v+val}val if v set at all, else emptynone

String / substring operations

ExpansionResult
${#v}Length of v in characters
${v:n}Substring from position n to end
${v:n:len}Substring: len chars from position n
${v: -n}Last n characters (space before minus)
${v#pat}Remove shortest prefix matching pat
${v##pat}Remove longest prefix matching pat
${v%pat}Remove shortest suffix matching pat
${v%%pat}Remove longest suffix matching pat
${v/pat/rep}Replace first match of pat with rep
${v//pat/rep}Replace all matches of pat with rep
${v/#pat/rep}Replace pat only at start
${v/%pat/rep}Replace pat only at end
${v^}Uppercase first char bash 4.0+
${v^^}Uppercase all chars bash 4.0+
${v,}Lowercase first char bash 4.0+
${v,,}Lowercase all chars bash 4.0+

Indirection and transformation

ExpansionResult
${!v}Value of the variable whose name is stored in v
${!prefix@}Names of all variables starting with prefix
${v@Q}Shell-quoted form of v bash 4.4+
${v@A}declare statement to recreate v bash 4.4+
${v@a}Attribute flags of v (r=readonly, i=integer, a=array…) bash 4.4+
${v@E}Expand escape sequences in v bash 4.4+
declare -n ref=nameMake ref an alias for the variable named name bash 4.3+

✏️ Exercises

Each exercise is designed to be solved purely with parameter expansion — resist the urge to reach for sed, awk, or python. The goal is to develop fluency with the shell's built-in string machinery.

Exercise 1
Write a function called path_info() that accepts a full file path as its only argument and — using only parameter expansion, no external commands — prints four lines: the directory, the filename (with extension), the base name (without any extension), and the extension (without the dot). Test it with /var/log/nginx/access.log.2.
Hint: directory = ${path%/*}, filename = ${path##*/}, extension = ${filename##*.}, base = ${filename%%.*}. Edge case: a file with no extension.
Sample Solution
#!/usr/bin/env bash set -euo pipefail path_info() { local path="${1:?path_info: argument required}" local dir filename base ext dir="${path%/*}" filename="${path##*/}" # Only split extension if there is a dot in the filename if [[ "$filename" == *"."* ]]; then ext="${filename##*.}" base="${filename%%.*}" else ext="" base="$filename" fi printf "Directory : %s\n" "$dir" printf "Filename : %s\n" "$filename" printf "Base name : %s\n" "$base" printf "Extension : %s\n" "${ext:-(none)}" } path_info "/var/log/nginx/access.log.2" Directory : /var/log/nginx Filename : access.log.2 Base name : access Extension : log.2 path_info "/usr/bin/bash" Directory : /usr/bin Filename : bash Base name : bash Extension : (none)
Exercise 2
Write a function called return_multiple() that calculates the quotient and remainder of two integers (passed as arguments), and returns both values to the caller using a nameref — so the caller can use them as separate variables. Also write a second function swap() that uses two namerefs to swap the values of two caller-side variables in place, without using a temporary variable visible to the caller.
Hint: for return_multiple, accept two nameref names as the first two arguments. For swap, use local -n _a="$1" _b="$2" and a local temp variable for the swap itself.
Sample Solution
#!/usr/bin/env bash set -euo pipefail divmod() { # Usage: divmod QUOT_VAR REM_VAR DIVIDEND DIVISOR local -n _quot="$1" local -n _rem="$2" local _a="$3" _b="$4" [[ "$_b" -ne 0 ]] || { echo "divmod: division by zero" >&2; return 1; } _quot=$(( _a / _b )) _rem=$(( _a % _b )) } swap() { # Usage: swap VAR1 VAR2 — swaps values in place local -n _x="$1" local -n _y="$2" local _tmp="$_x" _x="$_y" _y="$_tmp" } # Test divmod declare -i q r divmod q r 17 5 echo "17 ÷ 5 = $q remainder $r" 17 ÷ 5 = 3 remainder 2 # Test swap x="hello"; y="world" echo "Before: x=$x y=$y" swap x y echo "After: x=$x y=$y" Before: x=hello y=world After: x=world y=hello
Exercise 3
Write a script called config_check.sh that uses ${var:?} to validate a set of required environment variables (APP_ENV, DB_HOST, DB_PORT, SECRET_KEY). For optional variables (LOG_LEVEL, TIMEOUT), provide sensible defaults using ${var:-default}. Then print a startup summary showing each variable's name, its value, and its source ("required", "default", or "env"). Use ${!prefix@} to discover and print all APP_ and DB_ variables automatically.
Hint: store the "source" of each optional variable by checking before applying the default: if [[ -n "${LOG_LEVEL+x}" ]]; then src="env"; else src="default"; fi. Then apply the default. For the prefix listing, loop over "${!APP_@}" and "${!DB_@}".
Sample Solution
#!/usr/bin/env bash # config_check.sh set -uo pipefail # no -e so :? error message is visible before exit # ── Required variables — script exits here if any are unset/empty ── APP_ENV="${APP_ENV:?APP_ENV is required (e.g. production, staging, dev)}" DB_HOST="${DB_HOST:?DB_HOST is required}" DB_PORT="${DB_PORT:?DB_PORT is required}" SECRET_KEY="${SECRET_KEY:?SECRET_KEY is required}" # ── Optional variables — track source before applying default ── resolve_optional() { local -n _var="$1" local _default="$2" if [[ -n "${_var+x}" && -n "${_var}" ]]; then echo "env" else _var="$_default" echo "default" fi } ll_src=$(resolve_optional LOG_LEVEL "INFO") to_src=$(resolve_optional TIMEOUT "30") # ── Print startup summary ───────────────────────────────────── echo printf "\033[1mConfiguration Summary\033[0m\n" printf '%.0s─' {1..50}; echo printf "%-18s %-25s %s\n" "Variable" "Value" "Source" printf '%.0s─' {1..50}; echo # Required for v in APP_ENV DB_HOST DB_PORT; do # Mask SECRET_KEY display="${!v}" printf "%-18s %-25s \033[32mrequired\033[0m\n" "$v" "$display" done # Show masked secret masked="${SECRET_KEY:0:4}****${SECRET_KEY: -2}" printf "%-18s %-25s \033[32mrequired\033[0m\n" "SECRET_KEY" "$masked" # Optional printf "%-18s %-25s \033[33m%s\033[0m\n" "LOG_LEVEL" "$LOG_LEVEL" "$ll_src" printf "%-18s %-25s \033[33m%s\033[0m\n" "TIMEOUT" "$TIMEOUT" "$to_src" printf '%.0s─' {1..50}; echo echo echo "All APP_ and DB_ variables in scope:" for v in "${!APP_@}" "${!DB_@}"; do printf " %s=%s\n" "$v" "${!v}" done
Exercise 4
Write a function called safe_log_args() that takes any number of arguments (as a command and its arguments would be passed to it), and logs them in a copy-pasteable form using ${arg@Q} to handle values containing spaces, quotes, or special characters. Then write a wrapper function run() that calls safe_log_args before executing its arguments as a command. Demonstrate it wrapping cp and mkdir with tricky paths.
Hint: in safe_log_args(), loop over "$@" and collect "${arg@Q}" into an array, then print the array joined with spaces. In run(), call safe_log_args "$@" first, then "$@" to execute. Test with paths containing spaces.
Sample Solution
#!/usr/bin/env bash set -euo pipefail safe_log_args() { local quoted=() local arg for arg in "$@"; do quoted+=( "${arg@Q}" ) done # Print as a reproducible shell command printf '\033[36m[RUN]\033[0m %s\n' "${quoted[*]}" >&2 } run() { safe_log_args "$@" "$@" } # Demonstration TMPDIR=$(mktemp -d) trap 'rm -rf "$TMPDIR"' EXIT # These commands have spaces and special chars in paths run mkdir -p "${TMPDIR}/my project/src files" run touch "${TMPDIR}/my project/src files/main script.sh" run cp "${TMPDIR}/my project/src files/main script.sh" \ "${TMPDIR}/backup of main script.sh" [RUN] mkdir -p '/tmp/tmp.abc123/my project/src files' [RUN] touch '/tmp/tmp.abc123/my project/src files/main script.sh' [RUN] cp '/tmp/tmp.abc123/my project/src files/main script.sh' '/tmp/tmp.abc123/backup of main script.sh' # The log output is safe to copy-paste and re-run exactly