Network Operations from the Shell

🌐 Intermediate Topic 8 — Network Operations from the Shell

Shell scripts are first-class network citizens. curl alone covers the vast majority of HTTP work: downloading files, calling REST APIs, sending webhooks, checking service health, and posting alerts. Below HTTP, bash's built-in /dev/tcp pseudo-device handles raw TCP without any external tool. This chapter covers curl in production depth — headers, authentication, retries, rate limiting, parallel downloads — along with wget for mirroring, /dev/tcp for low-level probing, and the patterns used in real monitoring and deployment scripts.

1 — `curl` In Depth

curl is the workhorse. The flags you use every day are only a fraction of what it offers. Understanding its output model — where the body goes, where headers go, how to capture the HTTP status code — is what separates scripts that work from scripts that silently fail.

🌐 Core curl flags and output model

# ── Output control ────────────────────────────────────────────
curl -s   URL           # silent: suppress progress/errors
curl -S   URL           # show errors even when silent (-sS together)
curl -o FILE URL        # write body to FILE instead of stdout
curl -O   URL           # write to filename from URL
curl -L   URL           # follow redirects (3xx)
curl -I   URL           # HEAD only — show response headers, no body
curl -v   URL           # verbose: show all request and response headers

# ── Capture HTTP status code separately ───────────────────────
# -w writes a format string AFTER the response body
http_code=$(curl -sS -o /dev/null -w '%{http_code}' URL)

# Body + status in one call — split on the last newline
response=$(curl -sS -w $'\n%{http_code}' URL)
status="${response##*$'\n'}"
body="${response%$'\n'*}"

# ── Useful -w variables ───────────────────────────────────────
# %{http_code}      HTTP status (200, 404, etc.)
# %{time_total}     total elapsed seconds
# %{time_connect}   time to TCP connect
# %{size_download}  bytes downloaded
# %{speed_download} bytes/second
# %{redirect_url}   final URL after redirects
# %{content_type}   Content-Type header value

curl -sS -o /dev/null \
     -w 'status=%{http_code}  time=%{time_total}s  size=%{size_download}B\n' \
     https://example.com

# ── Timeouts ──────────────────────────────────────────────────
curl --connect-timeout 5   # max seconds to establish TCP connection
curl --max-time       30   # max total seconds for the whole transfer
# Always set both — without them curl hangs indefinitely

# ── Request methods and bodies ────────────────────────────────
curl -X GET    URL
curl -X POST   -d 'key=val'                URL
curl -X POST   -d '{"x":1}' \
               -H 'Content-Type: application/json'  URL
curl -X PUT    -d @payload.json \
               -H 'Content-Type: application/json'  URL
curl -X DELETE URL
curl -X PATCH  -d '{"field":"value"}' \
               -H 'Content-Type: application/json'  URL

# ── Headers ───────────────────────────────────────────────────
curl -H 'Accept: application/json'           URL
curl -H 'Authorization: Bearer TOKEN'        URL
curl -H 'X-API-Key: SECRET'                  URL
curl -H 'Authorization: Basic BASE64CREDS'   URL
# Or let curl encode basic auth:
curl -u 'user:password'                       URL

# ── Dump response headers to a file ──────────────────────────
curl -sSD headers.txt -o body.json URL
# -D FILE writes response headers; -o FILE writes body

# ── SSL / TLS ─────────────────────────────────────────────────
curl -k              URL   # skip cert verification (dev only)
curl --cacert ca.pem  URL   # custom CA bundle
curl --cert cl.pem \
     --key  cl-key.pem URL  # mutual TLS (client cert)

Retries and rate limiting

🌐 curl built-in retry and production-safe wrappers

# ── Built-in retry flags (curl 7.12+) ─────────────────────────
curl --retry           3    # retry up to N times on transient failures
curl --retry-delay     2    # seconds between retries
curl --retry-max-time  60   # total retry budget in seconds
curl --retry-connrefused    # also retry on connection refused
# curl retries on: network errors, 408, 429, 500, 502, 503, 504
# curl does NOT retry on 4xx errors by default

# ── Production-safe curl wrapper ──────────────────────────────
api_call() {
    # Usage: api_call METHOD URL [extra curl args...]
    local method="$1" url="$2"; shift 2
    local response status body

    response=$(curl -sS                    \
                    -X "$method"            \
                    --connect-timeout 10   \
                    --max-time       30   \
                    --retry          3    \
                    --retry-delay    2    \
                    --retry-connrefused    \
                    -w $'\n%{http_code}'   \
                    "$@"                    \
                    "$url")

    status="${response##*$'\n'}"
    body="${response%$'\n'*}"

    if [[ "$status" -ge 200 && "$status" -lt 300 ]]; then
        printf '%s' "$body"
        return 0
    fi

    printf 'HTTP %s from %s: %s\n' "$status" "$url" \
           "$(jq -r '.message // .error // "no message"' <<< "$body" 2>/dev/null || printf '%s' "$body")" >&2
    return 1
}

# Usage:
users=$(api_call GET https://api.example.com/users \
         -H "Authorization: Bearer $TOKEN")

api_call POST https://api.example.com/items \
         -H 'Content-Type: application/json' \
         -d '{"name":"widget"}'

# ── Manual rate limiting (honour 429 Retry-After) ─────────────
curl_rate_limited() {
    local url="$1"; shift
    local attempt

    for (( attempt=1; attempt<=5; attempt++ )); do
        local headers body status
        local tmphead; tmphead=$(mktemp)

        body=$(curl -sS -w $'\n%{http_code}' -D "$tmphead" "$@" "$url")
        status="${body##*$'\n'}"
        body="${body%$'\n'*}"

        if [[ "$status" == "429" ]]; then
            local retry_after
            retry_after=$(grep -i 'Retry-After:' "$tmphead" | \
                          awk '{print $2}' | tr -d $'\r')
            retry_after=${retry_after:-5}
            printf 'Rate limited — sleeping %ds (attempt %d/5)\n' \
                   "$retry_after" "$attempt" >&2
            sleep "$retry_after"
            rm -f "$tmphead"
            continue
        fi

        rm -f "$tmphead"
        printf '%s' "$body"
        [[ "$status" -ge 200 && "$status" -lt 300 ]]
        return
    done

    printf 'Rate limit exceeded after 5 attempts\n' >&2
    return 1
}

File transfer and parallel downloads

🌐 Downloading files, progress display, parallel fetches

# ── Download with progress bar ────────────────────────────────
curl -L --progress-bar -o output.tar.gz URL

# ── Resume an interrupted download ────────────────────────────
curl -C - -o partial.tar.gz URL   # -C - = auto-detect resume offset

# ── Download only if remote is newer (conditional GET) ────────
curl -z existing_file.txt -o existing_file.txt URL
# Sends If-Modified-Since based on local file's mtime

# ── Parallel downloads with xargs ─────────────────────────────
# Download 4 URLs concurrently
printf '%s\n' "${urls[@]}" | \
    xargs -P4 -I{} curl -sS -L -o '{}'.html '{}'

# ── Upload a file ─────────────────────────────────────────────
curl -F "file=@/path/to/upload.txt" URL          # multipart form
curl -T /path/to/file ftp://server/dest/         # FTP PUT
curl -X PUT --data-binary @file.bin URL          # binary body

# ── Send form data (URL-encoded) ──────────────────────────────
curl -d 'user=alice&pass=secret' URL
curl --data-urlencode 'msg=hello world' URL   # auto URL-encodes spaces

# ── Cookies ───────────────────────────────────────────────────
curl -c cookies.txt -b cookies.txt URL   # save + send cookies

# ── Use a config file to avoid repeating flags ────────────────
# ~/.curlrc or a project-specific file
cat > .curlrc <<'EOF'
silent
show-error
location
connect-timeout = 10
max-time = 60
retry = 3
EOF
curl --config .curlrc URL

2 — HTTP Health Checks and Service Monitoring

🌐 Health check patterns used in production scripts

# ── Simple: is the service up? ────────────────────────────────
http_ok() {
    local url="$1"
    local code
    code=$(curl -sS -o /dev/null -w '%{http_code}' \
              --connect-timeout 5 --max-time 10 "$url" 2>/dev/null)
    [[ "$code" -ge 200 && "$code" -lt 400 ]]
}

http_ok https://api.example.com/health && echo "up" || echo "down"

# ── Check multiple services and report status ─────────────────
check_services() {
    local -a endpoints=( "$@" )
    local ok=0 failed=0

    printf '%-45s %-6s %s\n' "ENDPOINT" "CODE" "TIME"
    printf '%.0s─' {1..65}; echo

    local url
    for url in "${endpoints[@]}"; do
        local result
        result=$(curl -sS -o /dev/null \
                      --connect-timeout 5 --max-time 10 \
                      -w '%{http_code} %{time_total}' \
                      "$url" 2>/dev/null \
                      || echo "000 0")
        local code="${result% *}"
        local time="${result#* }"

        local colour='\033[32m'   # green
        if   [[ "$code" == "000" ]]; then colour='\033[31m'; (( failed++ ))  # red
        elif (( code >= 400 ));      then colour='\033[31m'; (( failed++ ))
        elif (( code >= 300 ));      then colour='\033[33m'; (( ok++ ))     # yellow
        else                               (( ok++ ))
        fi

        printf "%-45s ${colour}%-6s\033[0m %ss\n" \
               "$url" "$code" "$time"
    done

    printf '\n%d OK, %d failed\n' "$ok" "$failed"
    (( failed == 0 ))
}

check_services \
    https://api.example.com/health \
    https://app.example.com/ping \
    https://db.example.com:8080/status

# ── Wait until a service becomes healthy (deploy pattern) ─────
wait_for_http() {
    local url="$1" timeout="${2:-60}"
    local deadline=$(( $(date '+%s') + timeout ))

    printf 'Waiting for %s (timeout: %ds)...' "$url" "$timeout"
    while (( $(date '+%s') < deadline )); do
        if http_ok "$url"; then
            printf ' ready\n'; return 0
        fi
        printf '.'
        sleep 2
    done
    printf ' TIMEOUT\n' >&2
    return 1
}

wait_for_http http://localhost:8080/health 120
echo "Service is up — starting smoke tests"

3 — Sending Notifications via HTTP APIs

🌐 Webhook notifications: Slack, Teams, generic HTTP

# ── Slack webhook ─────────────────────────────────────────────
slack_notify() {
    local webhook="${SLACK_WEBHOOK_URL:?SLACK_WEBHOOK_URL not set}"
    local message="$1"
    local colour="${2:-good}"   # good | warning | danger

    local payload
    payload=$(jq -n \
        --arg msg    "$message" \
        --arg colour "$colour" \
        '{attachments: [{text: $msg, color: $colour, mrkdwn_in: ["text"]}]}')

    curl -sS -X POST \
         -H 'Content-Type: application/json' \
         --connect-timeout 5 --max-time 15 \
         -d "$payload" "$webhook" > /dev/null
}

# ── PagerDuty / generic alert ─────────────────────────────────
pagerduty_alert() {
    local routing_key="${PD_ROUTING_KEY:?}"
    local summary="$1"
    local severity="${2:-error}"   # critical|error|warning|info

    local payload
    payload=$(jq -n \
        --arg key  "$routing_key" \
        --arg sum  "$summary"     \
        --arg sev  "$severity"    \
        --arg src  "$(hostname)"  \
        '{
          routing_key:  $key,
          event_action: "trigger",
          payload: {
            summary:  $sum,
            severity: $sev,
            source:   $src
          }
        }')

    curl -sS -X POST \
         -H 'Content-Type: application/json' \
         -d "$payload" \
         https://events.pagerduty.com/v2/enqueue
}

# ── Generic webhook dispatcher ────────────────────────────────
send_webhook() {
    local url="$1" payload="$2"
    local secret="${WEBHOOK_SECRET:-}"
    local extra_headers=()

    if [[ -n "$secret" ]]; then
        # HMAC-SHA256 signature (GitHub-style)
        local sig
        sig="sha256=$(printf '%s' "$payload" | openssl dgst -sha256 -hmac "$secret" -hex | awk '{print $2}')"
        extra_headers=( -H "X-Hub-Signature-256: $sig" )
    fi

    curl -sS -X POST \
         -H 'Content-Type: application/json' \
         "${extra_headers[@]}" \
         --connect-timeout 5 --max-time 15 \
         -d "$payload" "$url"
}

4 — `/dev/tcp`: Raw TCP Connections from Bash

Bash has a built-in virtual filesystem under /dev/tcp/HOST/PORT and /dev/udp/HOST/PORT. Opening a file descriptor to these paths establishes a TCP (or UDP) connection — no external tools needed. This is invaluable when you're on a minimal system where curl, netcat, and telnet aren't available.

🌐 /dev/tcp for port checks, HTTP, and raw protocol probes

# ── TCP port reachability check ───────────────────────────────
tcp_port_open() {
    local host="$1" port="$2" timeout="${3:-5}"
    # timeout command wraps the redirect to enforce deadline
    timeout "$timeout" bash -c \
        ">/dev/tcp/${host}/${port}" 2>/dev/null
}

tcp_port_open db.example.com 5432  && echo "DB reachable" || echo "DB unreachable"
tcp_port_open redis.internal  6379  && echo "Redis up"

# ── Raw HTTP request via /dev/tcp ─────────────────────────────
http_get_raw() {
    local host="$1" path="${2:-/}" port="${3:-80}"
    exec 3<>/dev/tcp/"${host}"/"${port}"
    printf 'GET %s HTTP/1.0\r\nHost: %s\r\nConnection: close\r\n\r\n' \
           "$path" "$host" >&3
    cat <&3
    exec 3>&-
}

# Use it like curl on a system without curl:
http_get_raw example.com / 80 | head -20

# ── Check if a service speaks a specific protocol ─────────────
smtp_banner() {
    local host="$1" port="${2:-25}"
    exec 3<>/dev/tcp/"${host}"/"${port}"
    read -r -t 5 banner <&3
    exec 3>&-
    printf '%s\n' "$banner"
}
smtp_banner mail.example.com     → 220 mail.example.com ESMTP Postfix

# ── Check multiple ports in a loop ────────────────────────────
scan_ports() {
    local host="$1"; shift
    local port
    printf 'Port scan: %s\n' "$host"
    for port in "$@"; do
        if tcp_port_open "$host" "$port" 2; then
            printf '  %-6s \033[32mOPEN\033[0m\n'  "$port"
        else
            printf '  %-6s \033[31mCLOSED\033[0m\n' "$port"
        fi
    done
}

scan_ports db.internal 22 80 443 5432 6379 27017

/dev/tcp is a bash-only feature — it doesn't exist as a real filesystem entry; the kernel never sees it. It's handled entirely by bash's redirection code. It won't work in sh, dash, or zsh without bash.

5 — `wget` for Mirroring and Recursive Downloads

wget and curl overlap heavily for simple downloads, but wget has built-in recursive site mirroring and a simpler interface for batch operations. It also handles retries more transparently for download-focused use cases.

🌐 wget key patterns

# ── Basic download ────────────────────────────────────────────
wget URL                      # download to current dir, show progress
wget -q   URL                  # quiet
wget -O outfile URL            # write to specific file
wget -P /dest/ URL             # write to directory

# ── Resume and retry ──────────────────────────────────────────
wget -c   URL                  # continue/resume partial download
wget --tries=5 URL             # retry up to 5 times
wget --wait=2 URL              # wait 2s between retries
wget --timeout=30 URL          # 30s total timeout

# ── Check existence without downloading body ──────────────────
wget -q --spider URL           # exit 0 if URL exists, 1 if not
wget -q --spider URL 2>/dev/null && echo "exists"

# ── Mirror a website ──────────────────────────────────────────
wget --mirror \
     --convert-links \       # rewrite links for local viewing
     --adjust-extension \    # add .html to pages
     --page-requisites \     # also download CSS/JS/images
     --no-parent \           # don't go up to parent dirs
     -P ./mirror/ URL

# ── Download a list of URLs from a file ───────────────────────
wget -i urls.txt -P downloads/

# ── wget vs curl: when to use which ──────────────────────────
# wget: recursive, mirroring, simple batch downloads
# curl: APIs, custom headers/methods, piping, complex auth

6 — Quick Reference

Task	Command
GET, body to stdout	`curl -sS URL`
GET, save to file	`curl -sS -L -o file URL`
GET, just the HTTP status	`curl -sS -o /dev/null -w '%{http_code}' URL`
POST JSON	`curl -sS -X POST -H 'Content-Type: application/json' -d '{}' URL`
POST JSON from file	`curl -sS -X POST -H 'Content-Type: application/json' -d @file.json URL`
Bearer token auth	`curl -H "Authorization: Bearer $TOKEN" URL`
Basic auth	`curl -u user:pass URL`
Follow redirects	`curl -L URL`
Set both timeouts	`curl --connect-timeout 5 --max-time 30 URL`
Retry 3 times	`curl --retry 3 --retry-delay 2 URL`
Response headers to file	`curl -D headers.txt -o body.json URL`
Resume download	`curl -C - -o file URL`
TCP port reachable?	`timeout 5 bash -c ">/dev/tcp/host/port"`
Raw GET via /dev/tcp	`exec 3<>/dev/tcp/host/80; printf 'GET / HTTP/1.0\r\n\r\n' >&3; cat <&3`
Check URL exists	`wget -q --spider URL`
Download list of URLs	`wget -i urls.txt -P dest/`
Mirror a site	`wget --mirror --convert-links --page-requisites URL`

✏️ Exercises

Exercise 1

Write a script called http_monitor.sh that reads a list of URLs from a file (one per line, lines starting with # are comments), checks each URL every N seconds (configurable via INTERVAL, default 60), and logs to a TSV file with columns: timestamp, URL, HTTP status, response time in ms, and UP/DOWN status. When a URL changes state (UP→DOWN or DOWN→UP) it should print a highlighted alert to stderr. Run until interrupted with Ctrl-C, handling the SIGINT cleanly.

Hint: maintain an associative array of last-known states. Use curl -w '%{http_code} %{time_total}'. Convert float seconds to ms with integer arithmetic after stripping the decimal. In the SIGINT trap, print a summary of how long each URL was down.

Sample Solution

#!/usr/bin/env bash
# http_monitor.sh URL_FILE [LOG_FILE]
set -euo pipefail

URL_FILE="${1:?Usage: http_monitor.sh URL_FILE [LOG_FILE]}"
LOG_FILE="${2:-http_monitor.tsv}"
INTERVAL=${INTERVAL:-60}

[[ -f "$URL_FILE" ]] || { printf 'Not found: %s\n' "$URL_FILE" >&2; exit 1; }

declare -A last_state=()
declare -A down_since=()
start_time=$(date '+%s')

# Write TSV header if file is new
[[ -f "$LOG_FILE" ]] || \
    printf 'timestamp\turl\thttp_code\tms\tstatus\n' > "$LOG_FILE"

check_url() {
    local url="$1"
    local ts; printf -v ts '%(%Y-%m-%d %H:%M:%S)T' -1
    local result
    result=$(curl -sS -o /dev/null \
                  --connect-timeout 5 --max-time 15 \
                  -w '%{http_code} %{time_total}' \
                  "$url" 2>/dev/null \
                  || echo "000 0")

    local code="${result% *}"
    local secs="${result#* }"
    # Convert float secs to int ms (remove decimal point, trim leading zeros)
    local ms; ms=$(printf '%.0f' "$(echo "$secs * 1000" | bc)")

    local status
    (( code >= 200 && code < 400 )) && status="UP" || status="DOWN"

    # Log to TSV
    printf '%s\t%s\t%s\t%s\t%s\n' \
           "$ts" "$url" "$code" "$ms" "$status" >> "$LOG_FILE"

    # State change detection
    local prev="${last_state[$url]:-UNKNOWN}"
    if [[ "$prev" != "$status" ]]; then
        if [[ "$status" == "DOWN" ]]; then
            printf '\033[31m[%s] DOWN: %s (HTTP %s)\033[0m\n' "$ts" "$url" "$code" >&2
            down_since["$url"]=$(date '+%s')
        else
            local outage=""
            [[ -v "down_since[$url]" ]] && \
                outage=" (was down for $(( $(date '+%s') - down_since[$url] ))s)"
            printf '\033[32m[%s] UP: %s%s\033[0m\n' \
                   "$ts" "$url" "$outage" >&2
            unset "down_since[$url]"
        fi
    fi
    last_state["$url"]="$status"
}

# Graceful shutdown
trap '
    printf "\n\033[1mMonitor stopped. Summary:\033[0m\n" >&2
    for u in "${!last_state[@]}"; do
        printf "  %-40s %s\n" "$u" "${last_state[$u]}" >&2
    done
    printf "Runtime: %ds\n" "$(( $(date +%s) - start_time ))" >&2
    exit 0
' INT TERM

# Main loop
printf 'Monitoring %s, interval=%ds, log=%s\n' "$URL_FILE" "$INTERVAL" "$LOG_FILE" >&2
while true; do
    while IFS= read -r url; do
        [[ "$url" =~ ^[[:space:]]*($|#) ]] && continue
        check_url "$url"
    done < "$URL_FILE"
    sleep "$INTERVAL"
done

Exercise 2

Write a function called download_with_verify() that downloads a file from a URL and verifies its integrity against a known checksum. It should accept the URL, the expected checksum, and the hash algorithm (sha256 or sha512) as arguments, write the file to a configurable DOWNLOAD_DIR, skip the download if the file already exists and matches the checksum, and clean up a partial download on failure. Print a clear success/failure message.

Hint: use curl -L -o "$dest" for the download. Use sha256sum or sha512sum to compute the hash of the downloaded file. Compare the first field of their output to the expected checksum. Use a trap inside the function to remove the partial file if interrupted.

Sample Solution

download_with_verify() {
    local url="${1:?url required}"
    local expected="${2:?expected checksum required}"
    local algo="${3:-sha256}"
    local destdir="${DOWNLOAD_DIR:-.}"

    local filename="${url##*/}"
    local dest="${destdir}/${filename}"

    local sum_cmd
    case "$algo" in
        sha256) sum_cmd="sha256sum" ;;
        sha512) sum_cmd="sha512sum" ;;
        md5)    sum_cmd="md5sum"    ;;
        *) printf 'Unknown algo: %s\n' "$algo" >&2; return 1 ;;
    esac

    _verify() {
        local actual
        actual=$("$sum_cmd" "$dest" | awk '{print $1}')
        [[ "$actual" == "$expected" ]]
    }

    if [[ -f "$dest" ]]; then
        if _verify; then
            printf '[skip] %s already exists and checksum matches\n' "$filename"
            return 0
        fi
        printf '[warn] %s exists but checksum mismatch — re-downloading\n' "$filename" >&2
        rm -f "$dest"
    fi

    mkdir -p "$destdir"

    # Cleanup partial download on any failure
    trap "rm -f '$dest'; printf '\033[31m[fail]\033[0m Download interrupted: %s\n' '$filename' >&2" \
         ERR RETURN INT

    printf '[down] %s → %s\n' "$url" "$dest"
    curl -fsSL \
         --connect-timeout 10 \
         --max-time 300       \
         --retry 3            \
         --retry-delay 2      \
         -o "$dest"           \
         "$url"

    if _verify; then
        printf '\033[32m[ok]\033[0m   %s (%s verified)\n' "$filename" "$algo"
        trap - ERR RETURN INT
    else
        local actual; actual=$("$sum_cmd" "$dest" | awk '{print $1}')
        printf '[fail] Checksum mismatch for %s\n  expected: %s\n  got:      %s\n' \
               "$filename" "$expected" "$actual" >&2
        rm -f "$dest"
        return 1
    fi
}

# Example usage:
DOWNLOAD_DIR=/tmp/downloads \
download_with_verify \
    https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 \
    af986793a515d500ab2d35f8d2aecd656e764504d683bd632d62991dc4f8eca0  \
    sha256

Exercise 3

Write a script called tcp_probe.sh that acts as a connectivity diagnostic tool. It should accept a host as its argument and then test: DNS resolution (using getent hosts or dig), TCP connectivity to a configurable list of ports (from PORTS env var, default 22 80 443), and an HTTP health check if port 80 or 443 is open. Use /dev/tcp for the port checks and output a structured report. All checks should have a configurable timeout defaulting to 3 seconds.

Hint: run DNS check first — if that fails, the port checks will also fail and you can report that. For each port check use timeout $TIMEOUT bash -c ">/dev/tcp/$host/$port". For HTTP, do a /dev/tcp GET and check for an HTTP response line. Use SECONDS or date +%s%N to measure each check's latency.

Sample Solution

#!/usr/bin/env bash
# tcp_probe.sh HOST
set -uo pipefail

HOST="${1:?Usage: tcp_probe.sh HOST}"
PORTS=( ${PORTS:-22 80 443} )
TIMEOUT=${TIMEOUT:-3}

ms_now() { date '+%s%N'; }

elapsed_ms() {
    local t0=$1
    printf '%d' $(( ($(ms_now) - t0) / 1000000 ))
}

print_result() {
    local label="$1" ok="$2" detail="${3:-}"
    if [[ "$ok" == "0" ]]; then
        printf '  \033[32m✓\033[0m %-25s %s\n' "$label" "$detail"
    else
        printf '  \033[31m✗\033[0m %-25s %s\n' "$label" "$detail"
    fi
}

printf '\033[1mTCP Probe: %s\033[0m\n\n' "$HOST"
printf 'DNS RESOLUTION\n'

# ── DNS check ─────────────────────────────────────────────────
local t0; t0=$(ms_now)
local ip
if ip=$(getent hosts "$HOST" 2>/dev/null | awk '{print $1; exit}') && [[ -n "$ip" ]]; then
    print_result "DNS" 0 "$ip ($(elapsed_ms $t0)ms)"
else
    print_result "DNS" 1 "FAILED — cannot resolve $HOST"
    printf '\nAll port checks skipped (DNS failed).\n'
    exit 1
fi

# ── Port checks ───────────────────────────────────────────────
printf '\nPORT CHECKS\n'
local open_ports=()

for port in "${PORTS[@]}"; do
    t0=$(ms_now)
    if timeout "$TIMEOUT" bash -c \
           ">/dev/tcp/${HOST}/${port}" 2>/dev/null; then
        print_result "Port $port" 0 "OPEN ($(elapsed_ms $t0)ms)"
        open_ports+=( "$port" )
    else
        print_result "Port $port" 1 "CLOSED or filtered"
    fi
done

# ── HTTP check if 80 or 443 is open ───────────────────────────
local http_port=""
for p in "${open_ports[@]}"; do
    [[ "$p" == "80" || "$p" == "8080" ]] && { http_port="$p"; break; }
done

if [[ -n "$http_port" ]]; then
    printf '\nHTTP PROBE (port %s)\n' "$http_port"
    t0=$(ms_now)
    local banner=""
    exec 7<>/dev/tcp/"${HOST}"/"${http_port}"
    printf 'HEAD / HTTP/1.0\r\nHost: %s\r\nConnection: close\r\n\r\n' \
           "$HOST" >&7
    if read -r -t "$TIMEOUT" banner <&7; then
        print_result "HTTP response" 0 "${banner//$'\r'/} ($(elapsed_ms $t0)ms)"
    else
        print_result "HTTP response" 1 "No response within ${TIMEOUT}s"
    fi
    exec 7>&-
elif for p in "${open_ports[@]}"; do [[ "$p" == "443" ]] && break; done; then
    printf '\nHTTP PROBE (443 — use curl for HTTPS)\n'
    t0=$(ms_now)
    local code
    code=$(curl -sS -o /dev/null -w '%{http_code}' \
              --connect-timeout "$TIMEOUT" --max-time "$((TIMEOUT*2))" \
              "https://${HOST}/" 2>/dev/null || echo "000")
    print_result "HTTPS" $(( code < 400 ? 0 : 1 )) "HTTP $code ($(elapsed_ms $t0)ms)"
fi

printf '\n'

Exercise 4

Write a script called api_poll.sh that polls a JSON API endpoint on a configurable interval, compares the response to the previous one, and logs changes. Usage: api_poll.sh URL FIELD where FIELD is a jq expression (e.g. .status or .data.count). When the value of FIELD changes, print a timestamped change record to stdout and optionally call a webhook (ALERT_WEBHOOK env var). After 10 consecutive failed requests, exit with an error.

Hint: store the previous value in a variable, initially empty (treat first run as baseline). Use jq -r "$FIELD" to extract the field. Track consecutive failures with a counter that resets on success. For the webhook, use curl -X POST -d "$payload" with a JSON body built by jq -n.

Sample Solution

#!/usr/bin/env bash
# api_poll.sh URL JQ_FIELD
set -euo pipefail

URL="${1:?Usage: api_poll.sh URL JQ_FIELD}"
FIELD="${2:?jq field expression required}"
INTERVAL=${INTERVAL:-30}
ALERT_WEBHOOK="${ALERT_WEBHOOK:-}"

prev_value=""
fail_count=0
readonly MAX_FAILS=10

printf 'Polling %s field=%s interval=%ds\n' "$URL" "$FIELD" "$INTERVAL" >&2
printf 'timestamp\told_value\tnew_value\n'  # TSV header

trap 'printf "\nStopped.\n" >&2; exit 0' INT TERM

while true; do
    local body status
    body=$(curl -sS -w $'\n%{http_code}' \
                --connect-timeout 5 --max-time 15 \
                "$URL" 2>/dev/null || echo $'null\n000')
    status="${body##*$'\n'}"
    body="${body%$'\n'*}"

    if [[ "$status" -lt 200 || "$status" -ge 400 ]]; then
        (( fail_count++ ))
        printf '[fail %d/%d] HTTP %s\n' "$fail_count" "$MAX_FAILS" "$status" >&2
        (( fail_count >= MAX_FAILS )) && {
            printf '%d consecutive failures — aborting\n' "$MAX_FAILS" >&2
            exit 1
        }
        sleep "$INTERVAL"
        continue
    fi
    fail_count=0

    local current
    current=$(jq -r "$FIELD" <<< "$body" 2>/dev/null || echo "PARSE_ERROR")

    if [[ -z "$prev_value" ]]; then
        # First run — just establish baseline
        printf '[baseline] %s = %s\n' "$FIELD" "$current" >&2
    elif [[ "$current" != "$prev_value" ]]; then
        local ts; printf -v ts '%(%Y-%m-%d %H:%M:%S)T' -1
        # Log change as TSV
        printf '%s\t%s\t%s\n' "$ts" "$prev_value" "$current"

        # Optional webhook alert
        if [[ -n "$ALERT_WEBHOOK" ]]; then
            local payload
            payload=$(jq -n \
                --arg ts    "$ts"         \
                --arg url   "$URL"        \
                --arg field "$FIELD"      \
                --arg old   "$prev_value" \
                --arg new   "$current"   \
                '{ts:$ts, url:$url, field:$field, old:$old, "new":$new}')
            curl -sS -X POST \
                 -H 'Content-Type: application/json' \
                 --max-time 10 \
                 -d "$payload" "$ALERT_WEBHOOK" > /dev/null 2>&1 || true
        fi
    fi
    prev_value="$current"
    sleep "$INTERVAL"
done

Network Operations from the Shell

🌐 Intermediate Topic 8 — Network Operations from the Shell

1 — curl In Depth

Retries and rate limiting

File transfer and parallel downloads

2 — HTTP Health Checks and Service Monitoring

3 — Sending Notifications via HTTP APIs

4 — /dev/tcp: Raw TCP Connections from Bash

5 — wget for Mirroring and Recursive Downloads

6 — Quick Reference

✏️ Exercises

1 — `curl` In Depth

4 — `/dev/tcp`: Raw TCP Connections from Bash

5 — `wget` for Mirroring and Recursive Downloads