Regular Expressions in Bash

Chapter 9 — Regex in awk

The awk Program Model

awk is a complete data-processing language built around a simple loop: for every line of input, test each pattern — if it matches, run its action. Regex is the most common pattern type, and awk provides richer regex integration than either grep or sed because it can combine pattern matching with arithmetic, string functions, and formatted output.

Copy# Minimal example: print lines containing "error"
awk '/error/' app.log            # pattern only — default action is print
awk '/error/ { print }' app.log  # explicit print — identical result

# Multiple rules — each is tested independently against every line
awk '/ERROR/ { errors++ }
     /WARN/  { warns++ }
     END     { print "Errors:", errors, "  Warnings:", warns }' app.log

Fields and $0

awk automatically splits each input line into fields separated by the field separator FS (default: any whitespace). Regex is most powerful in awk when combined with field access.

Copy# Print only the timestamp and message from error lines
awk '/ERROR/ { print $1, $2, $4, $5, $6 }' app.log

# Custom field separator: parse colon-delimited /etc/passwd
awk -F: '/\/bin\/bash$/ { print $1 }' /etc/passwd  # users with bash shell

# FS can itself be a regex (gawk/mawk/nawk)
awk -F'[,;|]' '{ print $2 }' data.txt   # split on comma, semicolon, or pipe

# Set FS inside BEGIN (equivalent to -F)
awk 'BEGIN { FS = ":" } /root/ { print $1, $6 }' /etc/passwd

# NF: last field regardless of how many fields a line has
awk '{ print $NF }' file.txt          # last field of every line
awk '{ print $(NF-1) }' file.txt      # second-to-last field

Regex Operators in awk

Copy# ~ operator: test a specific field
awk -F, '$3 ~ /^[0-9]+$/ { print $1, $3 }' data.csv   # field 3 is all digits
awk -F: '$3 ~ /^[0-9]+$/ && $3+0 >= 1000 { print $1 }' /etc/passwd  # UID >= 1000

# !~ operator: exclude a field pattern
awk -F, '$2 !~ /^[A-Z]/ { print "Bad name:", $2 }' names.csv

# Negated line pattern
awk '!/^#/ && !/^$/' config.txt   # skip comment and blank lines

# Range pattern: print lines between markers (inclusive)
awk '/^START/,/^END/' file.txt

# Range pattern with action
awk '/^\[database\]/,/^\[/ { print }' config.ini  # print [database] section

# Dynamic regex: variable used as regex with ~
PATTERN="ERROR"
awk -v pat="$PATTERN" '$0 ~ pat { print }' app.log

# ~ inside an if statement
awk '{
    if ($1 ~ /^[0-9]{4}-[0-9]{2}-[0-9]{2}$/)
        print "date:", $1
    else
        print "not a date:", $1
}' file.txt

The match() Function

match(string, regex) searches for the regex anywhere in string, sets two special variables, and returns the position of the match (or 0 if no match).

Copy# match() + substr() to extract the matched text
echo "order #12345 placed" | awk '{
    if (match($0, /[0-9]+/))
        print "Order number:", substr($0, RSTART, RLENGTH)
}'
Order number: 12345

# Extract version number from a string
echo "nginx/1.24.0" | awk '{ match($0, /[0-9]+\.[0-9.]+/); print substr($0, RSTART, RLENGTH) }'
1.24.0

# Use match() return value as a boolean
awk 'match($0, /ERROR: (.+)/) { print RSTART, RLENGTH }' app.log

# Loop to extract all matches (one per call — advance string manually)
echo "foo123 bar456 baz789" | awk '{
    s = $0
    while (match(s, /[0-9]+/)) {
        print substr(s, RSTART, RLENGTH)
        s = substr(s, RSTART + RLENGTH)
    }
}'
123
456
789

match() with Capture Arrays (gawk)

In gawk, match() accepts a third argument — an array that is populated with capture groups, giving awk true regex capture support.

Copy# match(string, regex, array) — gawk only
# array[0] = full match, array[1] = group 1, array[2] = group 2, ...
echo "2026-06-11" | gawk '{
    if (match($0, /([0-9]{4})-([0-9]{2})-([0-9]{2})/, a))
        printf "year=%s month=%s day=%s\n", a[1], a[2], a[3]
}'
year=2026 month=06 day=11

# Parse a log line into components
echo "2026-06-11 14:32:00 ERROR database connection failed" | \
gawk '{
    if (match($0, /^([0-9-]+) ([0-9:]+) ([A-Z]+) (.+)/, a))
        printf "date=%s time=%s level=%s msg=%s\n", a[1], a[2], a[3], a[4]
}'
date=2026-06-11 time=14:32:00 level=ERROR msg=database connection failed

# Extract URL components
echo "https://api.example.com:8080/v2/users" | \
gawk '{
    match($0, /^(https?):\/\/([^:/]+)(:([0-9]+))?(\/.*)?$/, a)
    print "scheme:", a[1]
    print "host:  ", a[2]
    print "port:  ", (a[4] ? a[4] : "default")
    print "path:  ", a[5]
}'

sub() and gsub()

sub(regex, replacement, target) replaces the first match; gsub(regex, replacement, target) replaces all matches. If target is omitted, $0 is used. Both return the number of replacements made.

Copy# sub(): replace first match in $0
echo "aaa bbb aaa" | awk '{ sub(/aaa/, "xxx"); print }'
xxx bbb aaa

# gsub(): replace all matches in $0
echo "aaa bbb aaa" | awk '{ gsub(/aaa/, "xxx"); print }'
xxx bbb xxx

# target a specific field
echo "foo ERROR bar ERROR" | awk '-F " " { gsub(/ERROR/, "FAULT", $2); print }'
foo FAULT bar ERROR   # only field 2 was targeted

# & in replacement = entire match (same as sed)
echo "hello world" | awk '{ gsub(/[a-z]+/, "[&]"); print }'
[hello] [world]

# Escape & and \ in awk replacement strings
echo "foo" | awk '{ gsub(/foo/, "a\\&b"); print }'   # → a&b  (literal &)
echo "foo" | awk '{ gsub(/foo/, "a\\\\b"); print }'  # → a\b  (literal \)

# Use gsub return value: count replacements
awk '{
    n = gsub(/ERROR/, "FAULT")
    if (n > 0) print NR": replaced", n, "occurrence(s)"
}' app.log

# Strip HTML tags from a line
echo "<b>Hello</b> <i>World</i>" | awk '{ gsub(/<[^>]*>/, ""); print }'
Hello World

gensub() — Capture Groups in Replacement (gawk)

gensub(regex, replacement, how, target) is a gawk extension that supports \1–\9 back-references in the replacement string. Unlike sub()/gsub(), it returns the modified string rather than modifying in-place.

Copy# gensub() syntax: gensub(regex, replacement, how [, target])
# how = "g" for global, "1" for first, "2" for second, etc.

# Reformat date YYYY-MM-DD to DD/MM/YYYY using capture groups
echo "2026-06-11" | gawk '{
    print gensub(/([0-9]{4})-([0-9]{2})-([0-9]{2})/, "\\3/\\2/\\1", "g")
}'
11/06/2026

# Swap first and last name
echo "Smith, John" | gawk '{ print gensub(/([A-Za-z]+), ([A-Za-z]+)/, "\\2 \\1", 1) }'
John Smith

# gensub does not modify $0 — assign the result
awk '{
    new = gensub(/([0-9]+)/, "(\\1)", "g")  # wrap all numbers
    print new
}' file.txt

# Replace only the second occurrence (how = "2")
echo "cat and cat and cat" | gawk '{ print gensub(/cat/, "dog", 2) }'
cat and dog and cat

Practical awk Recipes

Log file analysis

Copy# Count events by log level with formatted output
awk '{
    if ($3 ~ /^(ERROR|WARN|INFO|DEBUG)$/) counts[$3]++
}
END {
    for (level in counts)
        printf "%-8s %d\n", level, counts[level]
}' app.log

# Extract slow queries: lines where duration field exceeds a threshold
awk -F, '$4 ~ /ms$/ { dur = $4+0; if (dur > 500) print $2, dur"ms" }' queries.csv

# Summarise HTTP status codes from access log
awk '{ match($0, /" [0-9]{3} /); code = substr($0, RSTART+2, 3); codes[code]++ }
END { for (c in codes) print c, codes[c] }' access.log

CSV and structured data

Copy# Print rows where email field looks valid
awk -F, '$3 ~ /^[[:alnum:]._%+-]+@[[:alnum:].-]+\.[[:alpha:]]{2,}$/ { print $1, $3 }' users.csv

# Extract key=value pairs from config lines
awk '/^[[:alpha:]]/ {
    match($0, /([^=]+)=(.*)/, a)
    printf "key=[%s] val=[%s]\n", a[1], a[2]
}' config.txt

# Reformat: turn "Last, First" CSV into "First Last" TSV
awk -F, 'NR>1 {                              # skip header
    name = gensub(/([^,]+), ([^,]+)/, "\\2 \\1", 1, $1)
    print name "\t" $2 "\t" $3
}' people.csv

Validation and reporting

Copy# Validate that every line of a file is a valid IPv4 address
awk '{
    if ($0 !~ /^([0-9]{1,3}\.){3}[0-9]{1,3}$/)
        print NR": not a valid IP:", $0
}' ip_list.txt

# Report lines where a field value is outside a numeric range
awk -F, '$2 ~ /^[0-9]+$/ && ($2+0 < 1 || $2+0 > 100) {
    print "Out of range on line", NR":", $2
}' scores.csv

# Find duplicate values in field 1
awk -F, '{ seen[$1]++ }
END { for (k in seen) if (seen[k] > 1) print "Duplicate:", k, "("seen[k]" times)" }' data.csv

Text transformation

Copy# Wrap long lines at word boundary (simple fold at 72 chars)
awk '{ while (length($0) > 72) {
    print substr($0, 1, 72)
    $0 = substr($0, 73)
} print }' longlines.txt

# Number every non-blank line
awk '/[^[:space:]]/ { printf "%4d  %s\n", ++n, $0; next } { print }' file.txt

# Convert Markdown-style headers to uppercase plain text
awk '/^#/ { gsub(/^#+[[:space:]]*/, ""); print toupper($0); next } { print }' README.md

awk Regex Flavour and Portability

Quick Reference — Chapter 9