Disk I/O: IOPS, Throughput & Latency
System Monitoring & Performance Diagnosis
Chapter 4 · Disk I/O: IOPS, Throughput & Latency
The machine from Chapters 1–3 turned out to be CPU-bound, with no meaningful disk involvement — its wa figure was 0.0 the whole time. This chapter is about the case Chapters 2 and 3 both mentioned but never demonstrated: a system where disk I/O genuinely is the bottleneck, using a fresh example — the nightly backup job Chapter 1 named as one of its four possible causes for "the app is slow."
Three Different Numbers, Three Different Questions
| Metric | The question it answers |
|---|---|
| IOPS | How many individual read/write operations happen per second |
| Throughput (MB/s) | How much total data moves per second |
| Latency | How long each individual operation takes to actually complete |
These three don't move together the way it might seem — a workload of many small, random operations (a busy database) is IOPS-bound and can look fine on throughput while genuinely struggling on IOPS and latency; a workload of a few large, sequential transfers (a backup job copying big files) is throughput-bound and can post huge MB/s numbers while barely touching IOPS. Knowing which kind of workload you're looking at changes which number actually matters.
iostat: Reading Linux's Own Disk Tool
r/s and w/s are IOPS (reads and writes per second); rkB/s and wkB/s are throughput; await is average latency in milliseconds, including time spent queued, not just time spent actually transferring data.
%util measures how much of the time the device had at least one request outstanding — not literally "how full" the disk is in any simple sense. A modern SSD can handle many requests in parallel through deep internal queuing, so it's entirely possible to see %util near 100% while the device is still comfortably keeping up with additional load. Treating %util alone as "the disk is maxed out" is the same category of mistake as reading a high load average without checking core count in Chapter 2.
The Metric That Actually Matters Most: Latency
await is usually the number closest to what a user actually experiences — a slow individual operation feels slow regardless of how busy the device looks overall or how much total throughput it's pushing. Rough reference points worth having in mind, since (per Chapter 1) a number alone means nothing without a baseline:
| Storage type | Typical latency |
|---|---|
| NVMe SSD | Well under 1 ms |
| SATA SSD | Low single-digit ms |
| Spinning HDD | 5–15 ms typical, often worse under random access due to physical seek time |
Busy vs. Slow: A Genuine Distinction
| %util | await | What it means |
|---|---|---|
| High | Low | Busy, but healthy — heavily utilized and keeping up comfortably |
| Low/moderate | High | Genuinely struggling — each request takes a long time even though the device isn't constantly occupied; often a queue-depth or hardware problem |
| High | High | Genuinely saturated — busy and slow at the same time, the clearest sign of a real bottleneck |
iotop: Finding Which Process Is Responsible
-o shows only processes actually doing I/O right now — a fast way to skip straight to the culprit rather than scanning every process on the system.
%util's counterpart), Avg. Disk sec/Read and Avg. Disk sec/Write (latency), and Disk Queue Length — plus a live, per-process breakdown of exactly which process is generating the read/write activity, the same job iotop does on Linux.
Working Example: Confirming Chapter 1's Backup-Job Scenario
Chapter 1 named "a nightly backup job saturating the disk" as one possible cause behind "the app is slow." A different server than Chapters 1–3's own machine shows exactly this: iostat -x 1 reports await at 85.40 ms — dramatically above any of this chapter's reference points for either SSD or spinning disk — alongside %util at 98.70%. High and high: genuinely saturated, not just busy. iotop immediately confirms the responsible process: an rsync backup job writing at over 38 MB/s, which today happens to be running well past its usual overnight window and directly overlapping with business-hours traffic. The fix isn't a code change or a resource upgrade — it's rescheduling or throttling the backup job so it no longer competes with live traffic for the same disk.
Hands-On Exercises
Explain why a busy database server and a backup job copying large files can both show heavy disk activity, yet be bottlenecked on two genuinely different metrics.
📄 View solutionExplain why this chapter says %util at 100% doesn't automatically mean a disk is overloaded, particularly for modern SSDs.
In this chapter's worked example, explain what specifically confirmed the disk was genuinely saturated (not just busy), and how iotop identified the actual cause.
Chapter 4 Quick Reference
- IOPS, throughput, and latency answer three different questions — a workload can be bound by one while looking fine on another
iostat -xshows all three at once;awaitis usually closest to what a user actually feels- %util isn't literally "how full" — modern SSDs with deep queuing can show near-100% while still comfortably keeping up
- Reference latencies: NVMe well under 1ms, SATA SSD low single-digit ms, HDD 5–15ms+
- High %util + low await = busy but healthy; low %util + high await = genuinely struggling; both high = genuinely saturated
iotop -ofinds the specific process responsible; Windows' Resource Monitor Disk tab does the same job- Next chapter: Disk Space: Filling Up, and Where It Actually Went