Capstone — Triaging Three Real Performance Tickets
System Monitoring & Performance Diagnosis
Chapter 10 · Capstone — Triaging Three Real Performance Tickets
Nine chapters built the pieces — a resource-first mindset, CPU, memory, disk, when it's not local at all, dashboards, process-level attribution, and telling a real trend apart from normal noise. This capstone applies all of it to three fresh tickets, worked more briskly than earlier chapters' own dedicated walkthroughs, since the underlying discipline should already feel familiar by now.
Ticket 1: "One API server in the pool is randomly slow"
One server out of an eight-machine pool serving the same API is intermittently slow; the other seven are fine. Started right after yesterday's deploy.
One machine (not the whole pool), sudden onset, correlating with a known event — yesterday's deploy. All four answers point toward something specific to this one machine's own current state, not a shared cause.
14.80 ÷ 8 cores ≈ 1.85 per core — genuinely overloaded, not a false alarm.
The process from yesterday's deploy — a new retry-handling code path with a bug that spins in a tight loop under a specific error condition, pegging a full core. Rolling back the deploy restores this server to the same behavior as the other seven.
Ticket 2: "The reporting service keeps crashing without warning"
A reporting service has been crashing sporadically for weeks, with no obvious error in its own application logs. Restarting it has become routine.
Not an unexplained crash at all — the kernel's own OOM killer, confirming memory exhaustion is the real cause.
Reviewing the dashboard's own historical data, checking available memory right after each restart, week over week:
| Week | Available memory, right after restart |
|---|---|
| 3 weeks ago | 2.1 GB |
| 2 weeks ago | 1.4 GB |
| Last week | 0.6 GB |
| This week | OOM-killed before ever settling |
The post-restart floor has been climbing steadily worse for weeks — a genuine, slow memory leak, not a series of unrelated crashes. (A quick df -h also turns up several weeks of accumulated core dump files under /var/crash from the repeated crashes — Chapter 5's own territory, worth cleaning up, though not the actual root cause here.)
The routine restarts were never a fix — they were exactly Chapter 1's own restart-destroys-evidence warning, repeated nightly for weeks, hiding a real leak that now needs actual profiling and a code fix.
Ticket 3: "Checkout is intermittently slow, but nothing local looks wrong"
The checkout page is slow for some users at unpredictable times. CPU, memory, and disk on the app server all look completely healthy every time anyone checks.
Load average sits comfortably under 1 per core. Memory shows a healthy "available" figure with no active swapping. iostat shows low %util and low await — disk isn't it either.
Almost no change — unlike Chapter 6's own worked example, packet loss isn't the answer this time. Ruling this out is itself real progress, not a dead end.
This server's network interface is rated for roughly 1 Gbps (about 125,000 kB/s) — 118,000 kB/s is right at the edge of what the link can physically carry. Not packet loss, but genuine bandwidth saturation on this machine's own interface. A large scheduled data-sync job, meant to run overnight, has been drifting later each day and now overlaps with peak checkout traffic — competing for the same finite link capacity, exactly Chapter 1's own "does it correlate with a known event" question, answered by a job's schedule quietly drifting rather than a one-off deploy.
Chapter Attribution
| Technique used above | Source chapter |
|---|---|
| Scoping questions; the restart-destroys-evidence warning (Tickets 1, 2, 3) | Chapter 1 |
| Load average ÷ core count (Ticket 1) | Chapter 2 |
The OOM killer, journalctl -k (Ticket 2) | Chapter 3 |
iostat's %util/await ruling disk out (Ticket 3) | Chapter 4 |
df -h surfacing accumulated core dumps (Ticket 2) — not the primary finding here, but the same technique applies directly to a disk-space-specific ticket | Chapter 5 |
Local netstat -s check, ruled out this time; interface throughput as an extension (Ticket 3) | Chapter 6 |
| Reviewing dashboard historical data (Ticket 2) | Chapter 7 |
top -o %CPU finding the specific responsible process (Ticket 1) | Chapter 8 |
| The floor technique across successive weeks (Ticket 2) | Chapter 9 |
Honest Scope Note
- No deep dive into application-level profiling tools (flame graphs, language-specific profilers) — this course stops at "which process," not "which line of code"
- No container/Kubernetes-specific resource metrics (cgroups limits, pod-level throttling) — the underlying principles carry over, but the concrete tooling genuinely differs
- No capacity-planning methodology (forecasting, load testing) — Chapter 9 recognizes a genuine trend, but deciding how much capacity to add is a separate discipline
- No database-specific performance tuning (query plans, index design) — a real, deep topic in its own right, out of scope here
- No setting up monitoring/alerting infrastructure itself — this course reads dashboards someone else built, not building the monitoring stack
Hands-On Exercises
Explain how Ticket 1's four scoping answers (one machine, sudden onset, tied to a deploy) narrowed the investigation before top was ever run.
Explain why Ticket 2's routine restarts were "never a fix," and what specifically the floor technique revealed that a single dashboard glance wouldn't have.
📄 View solutionExplain why ruling out retransmissions in Ticket 3 was still useful progress, even though it wasn't the actual cause, and what genuinely new check found the real answer.
📄 View solutionChapter 10 Quick Reference — Course Complete
- Ticket 1: a single overloaded server, traced to a specific buggy process from yesterday's deploy — CPU, load average, and process-level tools working together
- Ticket 2: routine restarts masking a real memory leak, only visible by comparing the post-restart floor across several weeks
- Ticket 3: local resources genuinely clean, retransmissions genuinely clean too — the real cause was network interface bandwidth saturation from a drifting scheduled job
- The recurring theme across all ten chapters: check against a baseline, don't destroy the evidence, and let elimination — not a guess — point you at the answer
- This closes System Monitoring & Performance Diagnosis, 10/10 chapters — the third complete course under the Technical Support subject, alongside Logging & Log Analysis and Network Troubleshooting