Process-Level Diagnosis: Finding the Specific Culprit

System Monitoring & Performance Diagnosis

Chapter 8 · Process-Level Diagnosis: Finding the Specific Culprit

Chapters 2 through 7 established which resource is under pressure. This chapter is about the natural next question: which specific process is actually responsible. Usually straightforward — until the culprit has already finished running by the time anyone looks.

top and htop's Per-Process View

Sorting top by CPU (Shift+P) or memory (Shift+M) surfaces the current heaviest consumers directly. htop offers the same information with a friendlier, scrollable, color-coded view, including per-core usage bars.

$ top top - 09:14:02 up 5 days, 2:10, 1 user, load average: 1.20, 1.05, 0.98 Tasks: 198 total, 2 running, 196 sleeping, 0 stopped, 0 zombie %Cpu(s): 8.1 us, 2.0 sy, 0.0 ni, 89.5 id, 0.2 wa, 0.0 hi, 0.2 si, 0.0 st PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3312 postgres 20 0 842104 91200 18320 R 4.3 0.6 0:12.44 postgres

A perfectly calm-looking morning — nothing here explains a slowdown a colleague swears happened overnight.

A live snapshot can completely miss a short-lived culprit
top's default view shows a single moment. A process that spiked CPU heavily for a short burst and finished (or dropped back down) before anyone happened to look won't appear as a top consumer in a snapshot taken afterward — even though it was the actual cause of a preceding slowdown. This is the exact same underlying problem as Chapter 7's own averaging gotcha, applied to process attribution instead of a dashboard graph: a brief, severe event can be invisible to a check that only looks at "right now."

Sampling Over Time Instead of Trusting One Look

Two ways to avoid needing to have been watching at the exact right second:

# Capture a rolling snapshot log every second for a minute $ top -b -d 1 -n 60 > top_log.txt # sar logs historical per-interval CPU data automatically on many systems $ sar -u -f /var/log/sysstat/sa08 02:00:01 AM CPU %user %nice %system %iowait %steal %idle 02:10:01 AM all 92.10 0.00 5.40 0.10 0.00 2.40 02:20:01 AM all 6.20 0.00 2.10 0.05 0.00 91.65

sar's historical logging confirms a genuine 92% CPU spike at 2:10 AM — evidence that already existed before anyone thought to go looking for it, since the system had been logging it automatically the whole time.

Windows has both equivalents too
Resource Monitor's per-process view extends Task Manager's basic CPU/memory columns with granular per-process disk and network activity — the closest Windows counterpart to iotop's per-process disk breakdown from Chapter 4. For historical logging over time, Performance Monitor's Data Collector Sets can log counters continuously to a file for later review, the same role sar plays on Linux.

Finding Who, Not Just When

sar confirms when the spike happened, but not by itself which process caused it. A top -b log, if one was already running, or a targeted historical process-accounting tool, fills that gap:

$ grep -A4 "02:10:0" top_log.txt top - 02:10:03 up 5 days, 2:10, 0 users, load average: 6.80, 3.20, 1.50 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9981 postgres 20 0 1042104 210200 28320 R 88.2 1.4 0:41.02 postgres: reindex_job

A nightly database reindex job, running at exactly 02:10 — the same timestamp sar already flagged. Two independent sources of evidence, checked separately, agreeing on the same answer.

Closing the Loop on Chapter 1's Own "0 zombie" Field

Every top snapshot in this course, starting with Chapter 1's very first example, has quietly shown a Tasks: line ending in 0 zombie — never explained until now. A zombie (shown as <defunct> in ps) is a process that has already finished running, but whose exit status hasn't yet been collected by its parent process. It isn't consuming meaningful CPU or memory — its actual resources are already released — but a large accumulation of zombies can exhaust the system's process table, preventing new processes from starting at all. A single zombie is rarely worth investigating on its own; a steadily growing zombie count over time points at a parent process with a genuine bug in how it manages its children.

$ ps aux | grep defunct web 4021 0.0 0.0 0 0 ? Z 03:14 0:00 [node] <defunct>

Working Example: The 2 AM Spike Nobody Was Watching

A dashboard (Chapter 7) shows a nightly CPU spike around 2 AM, but by the time anyone checks the following morning, a live top looks completely calm — exactly this chapter's own opening scenario. sar -u confirms the exact timestamp: 02:10 AM, 92% user CPU. Cross-referencing a top -b batch log already being captured by a scheduled job identifies the responsible process precisely: a nightly postgres reindex job. Rescheduling it to a genuinely quiet window, or spreading the work out instead of running it all at once, resolves the recurring nightly spike — found entirely through historical evidence, without anyone needing to be awake and watching at 2 AM.

Hands-On Exercises

Exercise 1

Explain why a colleague's live top check the morning after an overnight slowdown might show nothing unusual, even if a real, severe spike genuinely happened.

📄 View solution
Exercise 2

Explain the difference between what sar -u tells you and what a top -b log tells you, and why this chapter uses both together in its worked example rather than just one.

📄 View solution
Exercise 3

Explain what a zombie process actually is, why a single zombie usually isn't worth investigating, and what a steadily growing zombie count would suggest instead.

📄 View solution

Chapter 8 Quick Reference

  • top/htop, sorted by CPU or memory, surface the current heaviest consumers directly
  • A live snapshot can miss a short-lived culprit entirely — the same underlying problem as Chapter 7's averaging gotcha, applied to processes instead of graphs
  • top -b batch logging and sar's automatic historical logging both let you look back after the fact, rather than needing to catch the moment live
  • Windows equivalents: Resource Monitor's per-process view, and Performance Monitor's Data Collector Sets for historical logging
  • A zombie process (<defunct>) has finished but hasn't been reaped by its parent — harmless alone, but a growing count can exhaust the process table
  • Next chapter: Sustained vs. Transient Load: When a Spike Isn't a Problem