Further Reading & Resources
Chapter 8 — Further Reading & Resources
The tools covered in this course — top, iostat, ss, vmstat, strace, sysctl — are the everyday instruments every Linux administrator should reach for first. But performance work goes deeper. This chapter introduces the next layer: kernel-level profiling with perf, modern tracing with eBPF and bpftrace, the most valuable books and resources in the field, and a complete one-liner cheat sheet covering every topic from all eight chapters.
What this chapter covers: perf stat, perf top, perf record, and flame graphs for CPU profiling. eBPF architecture and why it matters. bpftrace one-liners. BCC ready-made tools (execsnoop, opensnoop, biolatency, tcpconnect, and more). Brendan Gregg's resources. Essential books. Key man pages. Full diagnostic one-liner cheat sheet organised by topic.
perf — Kernel-Level CPU Profiling
perf is the Linux kernel's built-in profiling tool. Unlike top or htop — which show what processes are running — perf shows what those processes are doing at the CPU instruction level. It can tell you which functions inside a program are consuming CPU, whether the bottleneck is in your code or in a library, and even whether cache misses or branch mispredictions are the root cause of high CPU usage.
Installation:
apt install linux-perf linux-tools-common (Debian/Ubuntu) or dnf install perf (RHEL/Fedora). You may also need linux-tools-$(uname -r). Run as root or with CAP_SYS_ADMIN capability.
perf stat
Run a command and count hardware events
Runs a program and prints hardware event counts when it finishes: CPU cycles, instructions, cache references, cache misses, branch instructions, branch misses. Useful for comparing two implementations of the same function.
perf stat -e cache-misses,instructions ./myprogram
perf top
Live CPU profiling — like htop but shows functions
Shows a continuously updated list of the kernel functions and user-space functions consuming the most CPU cycles system-wide. The fastest way to answer "what is the CPU actually executing?" No recording needed. Press
a to annotate a function with its assembly.
perf record
Capture a CPU profile to a file
Records samples to
perf.data. Attach to a running process with -p PID, or profile a command directly. -g enables call graph recording (stack traces) — required for flamegraphs. perf record -g -p 8821 sleep 30 profiles PID 8821 for 30 seconds.
perf report
Browse a recorded profile interactively
Opens
perf.data in an interactive TUI. Shows samples sorted by overhead %. Expand a function to see its callers and callees. Press a to annotate with source code (if debug symbols are available). perf report --stdio for plain text output.
# Quick hardware event summary for the currently running system
$ perf stat -a sleep 5
Performance counter stats for 'system wide':
39,842,156,430 cycles # 7.97 GHz
28,391,004,221 instructions # 0.71 insn per cycle
1,203,441,882 cache-references
312,441,112 cache-misses # 25.96% of all cache refs
# 26% cache miss rate is high — suggests a memory-intensive workload with poor locality
# Profile a running process for 30 seconds with call graphs
$ perf record -g -p 8821 sleep 30
[ perf record: Woken up 12 times to write data ]
[ perf record: Captured and wrote 3.241 MB perf.data (84241 samples) ]
$ perf report --stdio | head -30
# Overhead Command Shared Object Symbol
32.41% python3 libc-2.31.so [.] malloc
18.72% python3 python3.9 [.] _PyObject_Malloc
11.44% python3 python3.9 [.] dict_lookup
# malloc and _PyObject_Malloc together = 51% of CPU time in memory allocation.
# This process is spending more time allocating memory than doing actual work.
Flame Graphs — visualising the whole call stack at once
A flame graph collapses thousands of perf samples into a single SVG. Each box is a function. Width represents time spent. The call stack grows upward — a wide box at the top means that function is directly consuming CPU. Narrow boxes below it are its callers.
████████████████ malloc (32%) ██████████ dict_lookup (11%)
████████████████████████████ _PyObject_Malloc (19%) ███████████
████████████████████████████████████████████████████████████████████
process_request() — all paths lead through here (85%)
████████████████████████████████████████████████████████████████████
main() — entry point (100%)
Width = time. Height = call depth. Wide flat top = hot function. Narrow spike = deep call chain.
Read bottom-up: where your code spends time bubbles to the top.
# Generate a flame graph (requires Brendan Gregg's FlameGraph scripts)
$ git clone https://github.com/brendangregg/FlameGraph
$ perf record -F 99 -g -p 8821 sleep 30
$ perf script | ./FlameGraph/stackcollapse-perf.pl | ./FlameGraph/flamegraph.pl > flame.svg
$ xdg-open flame.svg # or copy to a web server and view in browser
# Off-CPU flame graph — shows time spent WAITING (not on CPU)
# Useful for I/O latency, lock contention, sleep() calls
$ perf record -e sched:sched_stat_sleep -e sched:sched_switch -g -p 8821 sleep 30
$ perf inject --stats -i perf.data | ./FlameGraph/stackcollapse-perf.pl --offcpu | ./FlameGraph/flamegraph.pl --color=io > offcpu.svg
eBPF and bpftrace — Modern Kernel Tracing
eBPF (extended Berkeley Packet Filter) allows small, safe programs to run inside the Linux kernel in response to events — system calls, function calls, network packets, scheduler events — without modifying kernel source code or loading a kernel module. The kernel verifies the program before running it, guaranteeing it can't crash the system. This makes eBPF the foundation of modern Linux observability.
bpftrace is a high-level scripting language for eBPF — it compiles one-liners and short scripts into eBPF programs and attaches them to kernel and user-space events. Think of it as awk for the kernel.
# Install bpftrace
$ apt install bpftrace # Debian/Ubuntu 20.04+
$ dnf install bpftrace # RHEL 8+
# List available probepoints (tracepoints + kprobes + uprobes)
$ bpftrace -l | head -30
tracepoint:syscalls:sys_enter_read
tracepoint:syscalls:sys_enter_write
kprobe:tcp_sendmsg
# ── One-liners ──────────────────────────────────────────────────
# Count system calls by process name (for 10 seconds)
$ bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); } interval:s:10 { print(@); clear(@); exit(); }'
# Trace every file opened on the system (filename + PID)
$ bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s opened %s\n", comm, str(args->filename)); }'
# Distribution of read() sizes — shows if reads are small (many syscalls) or large
$ bpftrace -e 'tracepoint:syscalls:sys_exit_read /args->ret > 0/ { @bytes = hist(args->ret); }'
# Disk I/O latency histogram (microseconds)
$ bpftrace -e 'tracepoint:block:block_rq_issue { @start[args->dev, args->sector] = nsecs; }
tracepoint:block:block_rq_complete /@start[args->dev, args->sector]/
{ @us = hist((nsecs - @start[args->dev, args->sector]) / 1000);
delete(@start[args->dev, args->sector]); }'
# TCP connection attempts by destination port
$ bpftrace -e 'kprobe:tcp_connect { @[((struct sock *)arg0)->__sk_common.skc_dport] = count(); }'
BCC — Ready-Made eBPF Tools
BCC (BPF Compiler Collection) ships with dozens of production-ready tracing tools. Install with apt install bpfcc-tools or dnf install bcc-tools. On Ubuntu the binaries are suffixed -bpfcc.
execsnoop
Every new process spawned on the system in real time — PID, parent, full command line. Invaluable for spotting rogue cron jobs or compiler spam.
opensnoop
Every file open() call — which process opened which file. Filter by PID with
-p PID. Great for finding what config files an application actually reads.biolatency
Block I/O latency histogram. Shows the distribution of disk request latencies in microseconds — immediately reveals if a disk is spiking to high latencies.
biotop
Like iotop but uses eBPF — top processes by disk I/O, updated live. Lower overhead than iotop. Shows reads and writes separately.
tcpconnect
Every outgoing TCP connection — destination IP and port. Shows you what a process is connecting to without a network capture.
tcpaccept
Every incoming TCP connection accepted — source IP, destination port, PID. Useful for auditing what services are accepting connections.
tcpretrans
TCP retransmissions in real time — source, destination, state. Retransmissions are a smoking gun for network packet loss.
cachetop
Page cache hit rate by process — how much of each process's I/O is served from the page cache vs going to disk.
runqlat
CPU run-queue latency histogram — how long processes wait on the run queue before getting CPU time. High latency here means CPU saturation.
profile
CPU profiler (like perf top) using eBPF — samples the call stack at 99Hz and reports the top functions. Lower overhead than perf for long-running profiles.
memleak
Tracks outstanding memory allocations — surfaces objects that were malloc'd but never freed. Run against a suspect process to confirm a memory leak.
funclatency
Latency histogram for any kernel or user-space function.
funclatency vfs_read shows how long filesystem reads take.
eBPF kernel version requirement: bpftrace needs kernel 4.9+; most BCC tools work best on 4.18+; advanced features (BTF, CO-RE) require 5.8+. On older kernels, fall back to the traditional tools covered in chapters 1–7. Check with
uname -r.
Brendan Gregg's Resources
Brendan Gregg (Senior Principal Engineer at Intel, formerly Netflix) is the most influential figure in Linux performance analysis. He created the USE method, the flamegraph visualisation, and the Linux Performance Observability Tools diagram — the field's most-referenced one-page reference.
- Linux Performance Observability Tools diagram — a single diagram showing which tool covers which layer of the Linux stack (hardware → kernel → user space), organised by CPU/memory/file/network/system call. Available at brendangregg.com/linuxperf.html. Print this and keep it on your desk.
- The USE Method — Utilisation / Saturation / Errors. For every resource, check if it's over-utilised, if there's a queue forming (saturation), and if there are error counts. Covered in Chapter 1 of this course.
- FlameGraph repository — github.com/brendangregg/FlameGraph — the Perl scripts used to generate flamegraph SVGs from perf, DTrace, or bpftrace data.
- brendangregg.com — extensive blog posts on perf, eBPF, and performance analysis methodology. The posts are long, detailed, and worth reading in full.
Essential Books
Essential
Systems Performance: Enterprise and the Cloud
The definitive reference on Linux and Solaris performance analysis. Covers every layer: CPU, memory, file systems, disks, network, and the cloud. The second half is an exhaustive reference for every tool and metric. If you buy one book on this topic, this is the one. Dense but comprehensive — read the chapter introductions, then use it as a reference for specific topics.
Essential
BPF Performance Tools
Companion to Systems Performance, focused entirely on eBPF, BCC, and bpftrace. Documents every BCC tool with examples and explains what each one is measuring at the kernel level. The definitive eBPF observability reference. Available free to read at the author's site.
Deep Dive
Linux Kernel Development
Explains how the Linux kernel works internally: the scheduler, virtual memory, the VFS layer, device drivers, synchronisation. Older (pre-cgroups/eBPF era) but the fundamentals haven't changed. Helps you understand why sysctl parameters have the effect they do.
Reference
The Linux Programming Interface
Comprehensive reference for Linux system calls and the C library — signals, processes, threads, files, sockets, memory mapping. Directly relevant when using strace output: knowing what each syscall does helps you interpret what a stuck process is waiting on. The man pages, explained.
Essential Man Pages
| Man Page | What it covers | Why it matters for performance work |
|---|---|---|
| man 5 proc | The /proc filesystem — every file under /proc/ documented | Explains what each field in /proc/meminfo, /proc/PID/status, /proc/stat, /proc/net/dev actually means. The authoritative source when a metric is ambiguous. |
| man sysctl.conf | Format and semantics of /etc/sysctl.conf and /etc/sysctl.d/ files | File format, ordering rules, how wildcards work, and which files override which. |
| man 5 limits.conf | PAM limits configuration — the format for /etc/security/limits.conf | All supported resource types (nofile, nproc, memlock, etc.) with valid value ranges. |
| man iostat | All iostat flags and column definitions | Precise definition of %util, await, aqu-sz, and the difference between -x and basic output. |
| man ss | Socket statistics — all filter expressions and output fields | The filter syntax for selecting sockets by state, address, or port. More powerful than the man page initially suggests. |
| man tc | Traffic control — queueing disciplines, classes, filters | Very large man page covering the full tc syntax for bandwidth shaping and throttling. |
| man perf | Linux perf tool — events, record options, report format | List of available hardware events (perf list) and all record/report flags. |
| man bpftrace | bpftrace language reference — probes, builtins, functions | Full language spec: all probe types (kprobe/uprobe/tracepoint/usdt), builtin variables (comm, pid, tid, nsecs), and map functions (count/hist/avg). |
| man 7 signal | Signal overview — all signal numbers, default actions, and semantics | Complete table of every signal, whether it can be caught/blocked, and what it does by default. Useful when you don't remember a signal number. |
| man free | free command output field definitions | Clarifies the "available" column vs "free" — a common source of confusion when reading memory output. |
Diagnostic One-Liner Cheat Sheet
The following one-liners cover the most common diagnostic tasks from all eight chapters. Keep this as a reference for the next time something is slow and you need to start somewhere.
CPU — Chapters 1 & 2
| uptime | Load average (1/5/15 min). Divide by nproc — above 1.0 = saturation |
| top -b -n 1 | head -20 | One-shot CPU snapshot sorted by %CPU (no interactive mode) |
| mpstat -P ALL 1 3 | Per-core CPU breakdown — spot single-core saturation hidden by averages |
| ps aux --sort=-%cpu | head -10 | Top 10 processes by CPU consumption right now |
| pgrep -c -f "pattern" | Count processes matching a full command-line pattern |
| ps faux | grep -A10 "script.sh" | Show process tree rooted at a specific process |
| renice +10 -p PID | Throttle a CPU-hungry process without killing it (nice +10 = lower priority) |
| taskset -cp 0,1 PID | Pin a process to CPU cores 0 and 1 to isolate it from other workloads |
| ps -o pid,stat,comm -p PID | Check if a process is D (uninterruptible), Z (zombie), or S (sleeping) |
Memory — Chapter 3
| free -h | Memory overview — read "available" not "free" for usable RAM |
| vmstat 1 5 | Memory + swap + I/O every second (5 samples) — si/so = swap in/out |
| grep -E "Mem|Swap|Dirty|Commit" /proc/meminfo | Key meminfo fields — committed memory, dirty pages, swap usage |
| ps aux --sort=-%mem | head -10 | Top 10 processes by RSS (actual RAM usage) |
| smem -r | head -10 | Top 10 by PSS (proportional memory — accounts for shared libraries correctly) |
| dmesg -T | grep -i "oom\|killed" | Check if the OOM killer has fired recently |
| watch -n 2 'grep -E "Dirty|Writeback" /proc/meminfo' | Monitor dirty page accumulation in real time |
| cat /proc/PID/oom_score | OOM kill priority for a specific process (0=immune, 1000=kill first) |
| sysctl vm.swappiness | Check current swap aggressiveness setting (default 60) |
Disk I/O & Storage — Chapter 4
| df -h | Disk space by filesystem — spot filesystems near 100% |
| df -i | Inode usage — "disk full" errors with free space = inode exhaustion |
| du -sh /* 2>/dev/null | sort -rh | head -15 | Find the largest top-level directories consuming space |
| iostat -x 1 5 | Disk I/O — watch %util, await, aqu-sz for saturation |
| iotop -o -b -n 3 | Which processes are doing I/O right now (requires root) |
| lsof | grep deleted | awk '$NF ~ /\(deleted\)/' | Files deleted but still held open (df vs du discrepancy) |
| find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -5 | Find directories with most files (inode exhaustion investigation) |
| journalctl --disk-usage | How much space systemd journal is consuming |
| truncate -s 0 /var/log/bigfile.log | Empty a log file without breaking open file handles |
Network — Chapter 5
| ss -tulpn | Listening TCP/UDP ports with process names (no DNS lookup) |
| ss -s | Socket summary — total counts by state (TIME_WAIT, ESTABLISHED, etc.) |
| ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn | Count TCP connections by state |
| ping -c 5 host && mtr --report host | Latency baseline then per-hop breakdown — spot where packet loss occurs |
| dig host +stats | grep "Query time" | DNS resolution time — is DNS adding latency? |
| curl -w "@curl-format.txt" -o /dev/null -s URL | HTTP timing breakdown: DNS / TCP connect / TLS / TTFB / total |
| nethogs eth0 | Bandwidth by process — which process is saturating the NIC |
| cat /proc/net/dev | column -t | NIC error counters — RX errors, dropped, overrun, TX errors |
| ethtool eth0 | grep -E "Speed|Duplex|Link" | NIC link speed and duplex — spot half-duplex mismatches |
Process Management — Chapter 6
| ps faux | Full process list with parent→child tree (forest view) |
| pstree -p PID | Process tree rooted at a specific PID with all PIDs shown |
| ps -o pid,ppid,pgid,stat,comm -p PID | Process details including parent PID and process group |
| kill -0 PID | Check if a PID exists without sending a real signal |
| pkill -TERM -f "full-command-pattern" | SIGTERM by full command line — safer than name-only match |
| kill -- -PGID | SIGTERM to entire process group (parent + all descendants) |
| cat /proc/PID/wchan | What kernel function a D-state process is stuck in |
| strace -p PID -T 2>&1 | head -20 | What system calls a running process is making (with timing) |
| lsof -p PID | wc -l | Count open file descriptors for a process |
| systemctl edit service | Create a drop-in systemd override (e.g. add LimitNOFILE=65536) |
Kernel Tuning — Chapter 7
| sysctl -a | grep "param" | Find kernel parameters matching a pattern |
| sysctl -w vm.swappiness=10 | Change a kernel parameter at runtime (lost on reboot) |
| sysctl --system | Reload all sysctl config files (simulate what happens at boot) |
| cat /proc/sys/fs/file-nr | System-wide open FD count: used / unused / max |
| cat /proc/PID/limits | grep "open files" | Process's actual open file limit (soft and hard) |
| ulimit -Hn | Hard open-files limit for the current shell session |
| grep -r "nofile" /etc/security/limits.d/ /etc/security/limits.conf | Find persistent ulimit configuration |
| tuned-adm recommend | Ask tuned which profile suits this hardware |
| sar -u -s 08:00 -e 12:00 | Historical CPU usage between 08:00 and 12:00 today |
| sar -r -f /var/log/sysstat/sa13 | Historical memory data from the 13th of the month |
Advanced Profiling — Chapter 8
| perf top | Live CPU profiler — which kernel/user functions are consuming cycles |
| perf stat -a sleep 5 | System-wide hardware event counts for 5 seconds |
| perf record -g -p PID sleep 30 && perf report | Record 30s CPU profile with call graphs and browse results |
| bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }' | Trace every file open on the system in real time |
| execsnoop-bpfcc | Show every new process spawned (BCC tool — requires bpfcc-tools) |
| biolatency-bpfcc | Block I/O latency histogram — reveals disk response time distribution |
| tcpconnect-bpfcc | Every outgoing TCP connection — destination IP and port |
| runqlat-bpfcc | CPU run-queue latency — how long processes wait to get CPU time |
| memleak-bpfcc -p PID 30 | Track memory allocations for 30 seconds — confirm a memory leak |
Where to go from here: The tools in this course are the foundation. The workflow is always the same — form a hypothesis about which resource is the bottleneck, use the relevant tool to confirm it, resolve the root cause, and verify the fix. perf and eBPF take you deeper when the standard tools don't show enough detail. Brendan Gregg's Systems Performance is the natural next step when you want to understand not just what the tools show, but why the Linux kernel behaves the way it does.