A ticket comes in: "the app is slow." One engineer restarts the service, waits to see if the complaints stop, and moves on. The other opens a resource monitor first, sees exactly what the system was doing in the moments before the restart would have wiped it away, and only then decides what to do. This course is entirely about becoming the second engineer — not because a restart never helps, but because checking first turns "it's slow" from a vague complaint into a specific, evidence-backed finding.

Four Things "Slow" Could Mean

"Slow" isn't one problem — it's a category covering at least four genuinely different resource bottlenecks, each with its own evidence and its own fix:

Notice the last row doesn't point at a chapter in this course at all — Chapter 6 covers exactly how to recognize that pattern and hand off cleanly, rather than re-teaching a course this site already has.

Scoping the Complaint First

Before touching any specific resource, a handful of questions narrow "it's slow" down dramatically — the same discipline this site's own netdiag1 course applies to connectivity complaints, adapted here for performance:

Why "Just Restart It" Is a Guess, Not a Diagnosis

A restart is tempting precisely because it often works — many resource problems do clear up, at least temporarily, once a process starts fresh. But "it worked" and "I know why it was happening" are different things entirely, and a restart that isn't preceded by a quick look at what the system was actually doing throws away the one chance to find out.

A Concrete Example: One Symptom, Several Different Causes

Take "the app is slow" and see how differently it resolves depending on which resource is actually behind it:

• A runaway process pegging every CPU core: a genuine CPU bottleneck — see Chapter 2
• The system swapping heavily to disk: a memory shortage severe enough that the OS is paging active memory out — see Chapter 3
• A nightly backup job saturating the disk: a disk I/O bottleneck that happens to overlap with business hours today — see Chapter 4
• Everything local looks perfectly healthy: the actual cause is very likely on the network path, not this machine at all — see Chapter 6, which hands off to netdiag1

Four genuinely different root causes, all hiding behind the same two words. Scoping the complaint first, then checking the right resource, is what tells them apart.

A Quick First Look

Before diving deep into any single resource, a quick overall check is worth running first — top on Linux, Task Manager or Resource Monitor on Windows. You'll learn to properly interpret every line of this in Chapters 2 and 3, but even without that depth yet, notice how much is visible in one glance:

A load average of 8.42, very little "free" memory, and heavy swap usage are all visible in this one snapshot — but they don't carry equal weight, and this chapter deliberately isn't explaining why yet. Chapter 2 covers what actually makes a load average high or normal for a given machine, and Chapter 3 explains why the low "free" figure here is very likely a red herring while the swap usage is a genuine warning sign. This exact snapshot is worth remembering — both of the next two chapters come back to it directly.

What This Course Covers

Reading CPU load and utilization correctly, understanding what "used" memory actually means, telling a busy disk apart from a genuinely slow one, tracking down where disk space actually went, recognizing when "slow" is really a network problem in disguise, reading a monitoring dashboard's own summarized view, drilling from system-wide symptoms down to the one specific process responsible, and telling a real trend apart from a normal, temporary spike. The capstone applies all of it to three realistic performance tickets.

Hands-On Exercises

Chapter 1 left one number deliberately unexplained: a load average of 8.42 in that chapter's top snapshot. This chapter explains exactly what load average measures (and what it surprisingly includes), why it means nothing at all without knowing the machine's core count, and how to read the rest of the CPU line — including two fields, wa and st, that can mean "slow" isn't really a CPU problem at all.

Load Average vs. CPU Utilization: Two Different Numbers

CPU utilization (the percentage figures in top's %Cpu(s) line) describes how busy the CPU is right now, over a very short interval. Load average is a different measurement entirely: a running average, over 1, 5, and 15 minutes, of the number of processes that were either using the CPU or waiting for something.

What "Normal" Load Actually Depends On: Core Count

A load average of 8 means something completely different on a 4-core machine than on a 16-core one — the raw number is meaningless without knowing how many cores are actually available to share the work.

Chapter 1's own example machine has 4 cores. A load average of 8.42 divided by 4 is just over 2.1 per core — meaning, on average, roughly twice as much work wants to run as the machine can actually handle at once. That's a genuine, meaningful overload signal, not a false alarm.

The %Cpu(s) Line In Depth

Chapter 1's snapshot also showed: 87.3 us, 9.1 sy, 0.0 ni, 2.1 id, 0.0 wa, 0.0 hi, 1.5 si, 0.0 st. Each field means something specific:

In Chapter 1's snapshot, wa and st are both 0.0, and us alone accounts for 87.3% — confirming this genuinely is a CPU-bound problem, not a disk bottleneck wearing a CPU-shaped disguise, and not a cloud tenant losing CPU time to a noisy neighbor.

Run Queue: Processes Actually Waiting for a Turn

vmstat shows a more direct number: the r column, the count of processes literally waiting for a CPU core right now — a narrower, more specific signal than load average's broader, I/O-inclusive definition.

An r value consistently well above the core count (6–8, on a 4-core machine) across several samples confirms the same story load average already suggested: genuine, sustained CPU contention, not a brief one-off spike.

Working Example: Fully Reading Chapter 1's Snapshot

Put together: a load average of 8.42 on a 4-core machine is roughly 2.1 per core — genuinely overloaded. The %Cpu(s) line's 0.0 wa and 0.0 st rule out both disk I/O and hypervisor steal time as explanations. 87.3% us confirms the time is being spent running actual application code, not the kernel or interrupt handling. A follow-up vmstat check showing r consistently at 6–8 confirms this isn't a passing blip. Every piece of evidence agrees: this machine has a genuine CPU bottleneck, and the next step is identifying which specific process is responsible — Chapter 8's own territory.

Hands-On Exercises

Chapter 1's snapshot showed only 412.6 MiB "free" out of 16 GB total, alongside heavy swap usage — and deliberately left both unexplained. This chapter resolves both: why the low "free" figure is very likely a false alarm, and why the swap figure deserves real attention, though — as the closing worked example shows — not always for the reason it first appears to.

The Classic "No Free Memory" False Alarm

Linux treats unused RAM as wasted RAM. Rather than leaving memory sitting empty, the kernel fills it with a cache of recently-read disk data — the page cache — so a repeated read can be served instantly from memory instead of hitting the disk again. That cache shows up in buff/cache, and on any machine that's been running a while, it tends to consume most of whatever isn't already allocated to running processes.

Reading free -h Properly

When Memory Pressure Is Real: Swap

Swap is fundamentally different from cache. When physical RAM is genuinely insufficient, the kernel moves active, in-use memory pages out to disk to free up space — a process that's orders of magnitude slower than RAM itself, since it involves real disk I/O for data a process actually needs. Unlike reclaiming cache, swapping has a real, measurable performance cost.

The OOM Killer: When Memory Pressure Gets Severe Enough

If physical RAM and swap are both genuinely exhausted, Linux's Out-Of-Memory (OOM) killer steps in and picks a process to terminate outright, freeing its memory by force. A process that simply vanishes, with no crash log of its own and no obvious cause, is a classic OOM-kill symptom — confirmed by checking the kernel's own log:

Reading these entries correctly is exactly the log-reading discipline this site's own Logging & Log Analysis (log1) course covers in depth — this is one specific, high-value example of it, applied to a memory-pressure investigation specifically.

Windows' Own Memory Model

Windows doesn't expose the exact same free-vs-cache split. Task Manager shows figures like "In use," "Available," "Committed," and "Cached," alongside a "Standby" list — memory holding recently-used file data that can be reclaimed, conceptually similar to Linux's page cache. Task Manager's own "Available" figure already accounts for reclaimable standby memory, in the same spirit as Linux's "available" column, though the two aren't a precise one-to-one match.

Working Example: Fully Reading Chapter 1's Snapshot

Chapter 1 showed 412.6 MiB free, 4417.8 MiB buff/cache, and swap at 1890.3 of 2048.0 MiB used. The low "free" figure is exactly this chapter's false alarm — a healthy 4.3 GiB of reclaimable cache sits right next to it. The swap figure, at first glance, looks like a serious active problem. But checking whether it's actually active right now means returning to Chapter 2's own vmstat sample from this same machine:

si and so are both 0 across every sample — nothing is actively swapping in or out right now. Combined with Chapter 2's own finding of 0.0 wa, the picture is consistent and honest: this machine's high swap-used figure is a leftover from an earlier period of pressure, not an active crisis, and the actual, currently-active bottleneck is exactly what Chapter 2 already found — genuine CPU contention. Memory isn't this machine's real problem right now; it's a secondary artifact worth noting, not the headline finding.

Hands-On Exercises

The machine from Chapters 1–3 turned out to be CPU-bound, with no meaningful disk involvement — its wa figure was 0.0 the whole time. This chapter is about the case Chapters 2 and 3 both mentioned but never demonstrated: a system where disk I/O genuinely is the bottleneck, using a fresh example — the nightly backup job Chapter 1 named as one of its four possible causes for "the app is slow."

Three Different Numbers, Three Different Questions

These three don't move together the way it might seem — a workload of many small, random operations (a busy database) is IOPS-bound and can look fine on throughput while genuinely struggling on IOPS and latency; a workload of a few large, sequential transfers (a backup job copying big files) is throughput-bound and can post huge MB/s numbers while barely touching IOPS. Knowing which kind of workload you're looking at changes which number actually matters.

iostat: Reading Linux's Own Disk Tool

r/s and w/s are IOPS (reads and writes per second); rkB/s and wkB/s are throughput; await is average latency in milliseconds, including time spent queued, not just time spent actually transferring data.

The Metric That Actually Matters Most: Latency

await is usually the number closest to what a user actually experiences — a slow individual operation feels slow regardless of how busy the device looks overall or how much total throughput it's pushing. Rough reference points worth having in mind, since (per Chapter 1) a number alone means nothing without a baseline:

Busy vs. Slow: A Genuine Distinction

iotop: Finding Which Process Is Responsible

-o shows only processes actually doing I/O right now — a fast way to skip straight to the culprit rather than scanning every process on the system.

Working Example: Confirming Chapter 1's Backup-Job Scenario

Chapter 1 named "a nightly backup job saturating the disk" as one possible cause behind "the app is slow." A different server than Chapters 1–3's own machine shows exactly this: iostat -x 1 reports await at 85.40 ms — dramatically above any of this chapter's reference points for either SSD or spinning disk — alongside %util at 98.70%. High and high: genuinely saturated, not just busy. iotop immediately confirms the responsible process: an rsync backup job writing at over 38 MB/s, which today happens to be running well past its usual overnight window and directly overlapping with business-hours traffic. The fix isn't a code change or a resource upgrade — it's rescheduling or throttling the backup job so it no longer competes with live traffic for the same disk.

Hands-On Exercises

Chapter 4 covered disk performance — how fast a disk can service requests. This chapter covers a different, equally common problem: a disk running out of space entirely. The two headline tools, df and du, usually agree — until they don't, and the gap between them turns out to be this chapter's own most useful diagnostic finding.

df: The Quick Overview

df reports space usage at the filesystem level — fast, and a good first check, but it says nothing about which files or directories are actually responsible.

du: Finding Where the Space Actually Went

du walks the actual directory tree, adding up real file sizes, and is the natural next step once df has confirmed a filesystem is genuinely full.

The Classic df/du Mismatch

Adding up the du figures above comes to roughly 30 GB. df reported 49 GB used on the same filesystem. That's not a rounding error — it's a real, specific, and genuinely common phenomenon.

Finding the Phantom Space

lsof +L1 lists open files with a link count of 0 — files that have been deleted but are still held open by a running process, exactly this chapter's own missing gap. The SIZE/OFF column here, roughly 18 GB, accounts for almost the entire shortfall between df and du's totals.

Windows' Own Disk-Usage Story

Windows' built-in tooling for "where did my space actually go" is comparatively weak — Storage settings gives only a rough, high-level breakdown by category, not a real directory-by-directory view. Third-party tools like WinDirStat fill that gap with a genuine visual breakdown, similar in spirit to du. A related situation to the deleted-but-open-file gotcha can occur on Windows too — a file with an open handle can resist being fully reclaimed even after deletion — though NTFS's own file-locking model differs enough from Linux's link-count semantics that the two aren't a precise match.

When Disk Fills Completely

A filesystem that reaches 100% doesn't just stop growing gracefully — a database can refuse further writes, a web server can fail to log or even serve requests, and in severe cases the operating system itself can struggle, since some system services genuinely need a small amount of free space to operate at all. This is worth treating with real urgency once a filesystem is closing in on full, not just noted and revisited later.

Working Example: The Missing 18 GB

A ticket: /var is at 98% and climbing, but a directory walk only accounts for about 30 GB of the 49 GB df reports as used. lsof +L1 finds the answer directly: a Java application still holding open a deleted 18 GB log file — deleted by an overnight cleanup script that never told the running process to reopen a fresh one. Restarting the application (or, less disruptively, sending it a signal to reopen its log handles, if it supports one) releases the space immediately, confirmed by a fresh df -h showing usage drop back in line with what du was already reporting.

Hands-On Exercises

Chapters 2 through 5 covered CPU, memory, and disk — all local to the machine you're standing on. This chapter is deliberately short, and deliberately doesn't try to re-teach a topic this site already covers in full: it's about recognizing when "slow" isn't a local resource problem at all, running one quick local-side check before committing to that conclusion, and handing off cleanly to this site's own Network Troubleshooting (netdiag1) course once that's confirmed.

The Tell: Everything Local Looks Healthy

The single clearest signal that a symptom is network-related rather than a local resource problem: CPU, memory, and disk all check out clean using Chapters 2 through 5's own tools, and the complaint persists anyway. A machine with plenty of spare CPU, healthy memory, and fast disk I/O that's still "slow" from a user's point of view is very likely waiting on something outside itself — the network path to it, or the network path from it to something it depends on.

Two Genuinely Different Network Symptoms

A Quick Local Check Before Handing Off

Before fully escalating, one more local-machine check is worth running: TCP retransmissions, a rough but genuinely useful local-side signal that packets are actually being lost somewhere on the network.

The absolute number matters less than the trend — over 1,100 additional retransmitted segments in one minute, on a machine that isn't under heavy load per Chapters 2–5, is a real, active signal that packets are being lost somewhere on the path, not just background noise.

When to Hand Off to Network Troubleshooting

Once local resources are confirmed clean and retransmissions confirm real packet loss is actually occurring, the right next step is this site's own netdiag1 course, not a deeper local investigation — its own ladder (from that course's Chapter 2 onward) is built exactly for this:

• Confirming DNS resolution and basic reachability — netdiag1 Chapters 4–5
• Checking whether the specific port/service is actually reachable — netdiag1 Chapter 6
• Firewalls, proxies, VPNs, and NAT possibly altering the path — netdiag1 Chapters 7–8
• Reading the actual HTTP/TLS-level result once a connection is made — netdiag1 Chapter 9

Repeating any of that material here would just be a shallower copy of a course this site already has — this chapter's own job is recognizing the handoff point, not replacing the course on the other side of it.

Working Example: The Slow Checkout Page

A ticket: the checkout page is slow, intermittently, for some users. CPU (Chapter 2) sits comfortably under load; memory (Chapter 3) shows healthy "available" figures with no active swapping; disk I/O (Chapter 4) is unremarkable. Everything local genuinely looks fine. A quick netstat -s check, repeated a minute apart, shows retransmissions climbing steadily — real, active packet loss. Handing off to netdiag1's own techniques (a traceroute to the specific dependency the checkout flow calls) finds the actual cause: intermittent loss on the path to a third-party payment provider's own API, not anything on this application server at all. The checkout page isn't slow because of anything this machine is doing — it's slow because it's waiting on a lossy network path to someone else's service.

Hands-On Exercises

Chapters 2 through 6 all assumed direct access — SSHing in and running a live command. Plenty of real environments don't work that way: the first (and sometimes only) view you get is a pre-built dashboard — Grafana, CloudWatch, Azure Monitor, or similar. Reading one correctly is a genuinely different skill from reading a live terminal, with its own real gotchas.

The Genuine Gotcha: Averaging Hides Spikes

Dashboards commonly store and display metrics aggregated over an interval — a 1-minute or 5-minute average is typical, both for storage efficiency and for keeping graphs readable. That averaging can genuinely hide the exact spike that caused real, user-visible pain.

Most monitoring tools let you switch the aggregation function applied to the same underlying data — from avg to max — over the same time window. Re-checking a suspicious-looking flat graph with max instead of avg is often the single fastest way to confirm whether a real spike is hiding underneath a smoothed-over average.

Reading a Typical Dashboard Layout

Most performance dashboards follow a similar shape: separate panels for CPU, memory, disk, and network, each a time-series line graph across a selectable time range, usually with a hover tooltip showing the exact value at a specific point.

Percentiles: A Better Summary Than Average

For response-time and latency panels specifically, an average can look perfectly fine while a meaningful fraction of requests are genuinely suffering — the same underlying "summarization hides the extreme" problem as averaging over time, applied instead to averaging across requests.

An average response time of 200ms sounds healthy. A p99 of 8 seconds, on the very same data, reveals that 1% of users — potentially a meaningful number of real people, depending on traffic — are having a genuinely bad experience the average alone never showed.

The Real Value: Correlating Multiple Panels at Once

Worth stating plainly, not just as a list of gotchas: a dashboard's genuine strength over one-command-at-a-time CLI tools is viewing CPU, memory, disk, and network side by side on the same time axis, and visually spotting that several of them moved together at the exact same moment — a correlation that's much harder to notice checking each metric separately, at different times, with separate commands.

Working Example: Chapter 4's Backup Job, Viewed on a Dashboard

Chapter 4's backup-job disk saturation, viewed as a dashboard instead of a live iostat session: the disk I/O panel shows a sustained plateau across the overnight window — visually obvious, no averaging trick needed, since sustained saturation doesn't get smoothed away the way a brief spike does. But the CPU panel for the same window, shown as an average, looks unremarkable. Switching that one panel's aggregation from avg to max reveals brief CPU spikes coinciding with the busiest moments of the backup — moments an averaged view alone would have hidden completely, exactly this chapter's own opening warning, now demonstrated against a scenario this course has already fully diagnosed by other means.

Hands-On Exercises

Chapters 2 through 7 established which resource is under pressure. This chapter is about the natural next question: which specific process is actually responsible. Usually straightforward — until the culprit has already finished running by the time anyone looks.

top and htop's Per-Process View

Sorting top by CPU (Shift+P) or memory (Shift+M) surfaces the current heaviest consumers directly. htop offers the same information with a friendlier, scrollable, color-coded view, including per-core usage bars.

A perfectly calm-looking morning — nothing here explains a slowdown a colleague swears happened overnight.

Sampling Over Time Instead of Trusting One Look

Two ways to avoid needing to have been watching at the exact right second:

sar's historical logging confirms a genuine 92% CPU spike at 2:10 AM — evidence that already existed before anyone thought to go looking for it, since the system had been logging it automatically the whole time.

Finding Who, Not Just When

sar confirms when the spike happened, but not by itself which process caused it. A top -b log, if one was already running, or a targeted historical process-accounting tool, fills that gap:

A nightly database reindex job, running at exactly 02:10 — the same timestamp sar already flagged. Two independent sources of evidence, checked separately, agreeing on the same answer.

Closing the Loop on Chapter 1's Own "0 zombie" Field

Every top snapshot in this course, starting with Chapter 1's very first example, has quietly shown a Tasks: line ending in 0 zombie — never explained until now. A zombie (shown as <defunct> in ps) is a process that has already finished running, but whose exit status hasn't yet been collected by its parent process. It isn't consuming meaningful CPU or memory — its actual resources are already released — but a large accumulation of zombies can exhaust the system's process table, preventing new processes from starting at all. A single zombie is rarely worth investigating on its own; a steadily growing zombie count over time points at a parent process with a genuine bug in how it manages its children.

Working Example: The 2 AM Spike Nobody Was Watching

A dashboard (Chapter 7) shows a nightly CPU spike around 2 AM, but by the time anyone checks the following morning, a live top looks completely calm — exactly this chapter's own opening scenario. sar -u confirms the exact timestamp: 02:10 AM, 92% user CPU. Cross-referencing a top -b batch log already being captured by a scheduled job identifies the responsible process precisely: a nightly postgres reindex job. Rescheduling it to a genuinely quiet window, or spreading the work out instead of running it all at once, resolves the recurring nightly spike — found entirely through historical evidence, without anyone needing to be awake and watching at 2 AM.

Hands-On Exercises

Every earlier chapter has been about confirming a genuine problem. This one is the deliberate flip side — recognizing elevated resource usage that's actually normal, bursty, expected behavior, and telling it apart from a real, sustained trend that genuinely needs action. Chapter 1's own baseline theme (an 80% CPU batch server being perfectly normal) gets its fullest treatment here.

Transient Spikes: Normal, Bursty Behavior

Plenty of legitimate causes produce a real, visible spike that isn't a problem at all: application startup (populating caches, JIT warmup), a scheduled batch job, a genuine burst of real traffic (a marketing email going out, a product launch), and — in managed-memory languages — garbage collection pauses, a normal, expected part of how the runtime reclaims memory, not inherently a bug.

Sustained Trends: The Kind That Actually Needs Action

A genuine sustained trend looks different from a transient spike in one specific way: it doesn't fully return to where it started. Real, sustained growth usually means one of two things — organic growth (more real users, more real data, a genuine capacity question) or a resource leak (a bug causing gradual, unbounded consumption that's never actually released, even during quiet periods when load is low).

The "Floor" Technique: Comparing Quiet Periods, Not Peaks

The single most useful technique in this chapter: instead of watching the peaks, watch the low points — the quiet periods, night after night, week after week. A transient spike, no matter how dramatic, returns fully to the same floor each time. A real leak or genuine growth trend leaves a slightly higher floor after every cycle, even if the peaks themselves look similar day to day.

A Genuine Judgment Call: When to Escalate a Trend

Recognizing a real, sustained trend is necessary, but it isn't the same as knowing what to do about it. Whether a confirmed trend needs urgent action, a scheduled fix, or genuine capacity planning depends on context this course can't fully supply on its own — how close the trend is to a hard limit, how fast it's actually climbing, and whether it represents real, wanted growth or a bug that needs fixing. Recognizing the trend is this chapter's job; deciding the response is a separate judgment call, often above what a single support engineer decides alone.

Working Example: The Nightly Restart That Was Never a Fix

A dashboard shows memory climbing to a concerning 85% by the end of every day — but a nightly scheduled restart drops it back to a comfortable-looking 40% each time, and the pattern has repeated, unremarked-on, for weeks. This is worth pausing on directly: Chapter 1 warned that a restart destroys the evidence needed to diagnose a problem. Here's the longer-term version of that same warning — a nightly restart didn't just destroy evidence once, it's been quietly masking a real problem for so long that nobody remembers it's a problem at all.

Applying the floor technique — checking available memory at the exact same point each day, right after the restart, over several weeks — tells a very different story than the daily graph alone:

Even the "reset" state is degrading, week over week — a real, slow leak, one that the nightly restart schedule has been tolerating rather than fixing. Left alone, the same trend that's been climbing for weeks eventually reaches the point where a nightly restart is no longer enough to buy back the space, and the actual leak — never diagnosed, only postponed — finally causes a real incident.

Hands-On Exercises

Nine chapters built the pieces — a resource-first mindset, CPU, memory, disk, when it's not local at all, dashboards, process-level attribution, and telling a real trend apart from normal noise. This capstone applies all of it to three fresh tickets, worked more briskly than earlier chapters' own dedicated walkthroughs, since the underlying discipline should already feel familiar by now.

Ticket 1: "One API server in the pool is randomly slow"

One server out of an eight-machine pool serving the same API is intermittently slow; the other seven are fine. Started right after yesterday's deploy.

Ticket 2: "The reporting service keeps crashing without warning"

A reporting service has been crashing sporadically for weeks, with no obvious error in its own application logs. Restarting it has become routine.

The routine restarts were never a fix — they were exactly Chapter 1's own restart-destroys-evidence warning, repeated nightly for weeks, hiding a real leak that now needs actual profiling and a code fix.

Ticket 3: "Checkout is intermittently slow, but nothing local looks wrong"

The checkout page is slow for some users at unpredictable times. CPU, memory, and disk on the app server all look completely healthy every time anyone checks.

Chapter Attribution

Honest Scope Note

Hands-On Exercises