Reading Monitoring Dashboards: Grafana & Cloud Console Metrics at a Glance
System Monitoring & Performance Diagnosis
Chapter 7 · Reading Monitoring Dashboards: Grafana & Cloud Console Metrics at a Glance
Chapters 2 through 6 all assumed direct access — SSHing in and running a live command. Plenty of real environments don't work that way: the first (and sometimes only) view you get is a pre-built dashboard — Grafana, CloudWatch, Azure Monitor, or similar. Reading one correctly is a genuinely different skill from reading a live terminal, with its own real gotchas.
The Genuine Gotcha: Averaging Hides Spikes
Dashboards commonly store and display metrics aggregated over an interval — a 1-minute or 5-minute average is typical, both for storage efficiency and for keeping graphs readable. That averaging can genuinely hide the exact spike that caused real, user-visible pain.
Most monitoring tools let you switch the aggregation function applied to the same underlying data — from avg to max — over the same time window. Re-checking a suspicious-looking flat graph with max instead of avg is often the single fastest way to confirm whether a real spike is hiding underneath a smoothed-over average.
Reading a Typical Dashboard Layout
Most performance dashboards follow a similar shape: separate panels for CPU, memory, disk, and network, each a time-series line graph across a selectable time range, usually with a hover tooltip showing the exact value at a specific point.
Percentiles: A Better Summary Than Average
For response-time and latency panels specifically, an average can look perfectly fine while a meaningful fraction of requests are genuinely suffering — the same underlying "summarization hides the extreme" problem as averaging over time, applied instead to averaging across requests.
| Metric | What it tells you |
|---|---|
| Average | A single blended figure — easily dominated by the bulk of fast, ordinary requests, hiding a smaller group of genuinely slow ones |
| p50 (median) | The typical request's experience — half of all requests were faster than this, half slower |
| p95 / p99 | What the slowest 5% or 1% of requests actually experienced — often dramatically worse than the average, and exactly the group most likely to generate complaints |
An average response time of 200ms sounds healthy. A p99 of 8 seconds, on the very same data, reveals that 1% of users — potentially a meaningful number of real people, depending on traffic — are having a genuinely bad experience the average alone never showed.
The Real Value: Correlating Multiple Panels at Once
Worth stating plainly, not just as a list of gotchas: a dashboard's genuine strength over one-command-at-a-time CLI tools is viewing CPU, memory, disk, and network side by side on the same time axis, and visually spotting that several of them moved together at the exact same moment — a correlation that's much harder to notice checking each metric separately, at different times, with separate commands.
Working Example: Chapter 4's Backup Job, Viewed on a Dashboard
Chapter 4's backup-job disk saturation, viewed as a dashboard instead of a live iostat session: the disk I/O panel shows a sustained plateau across the overnight window — visually obvious, no averaging trick needed, since sustained saturation doesn't get smoothed away the way a brief spike does. But the CPU panel for the same window, shown as an average, looks unremarkable. Switching that one panel's aggregation from avg to max reveals brief CPU spikes coinciding with the busiest moments of the backup — moments an averaged view alone would have hidden completely, exactly this chapter's own opening warning, now demonstrated against a scenario this course has already fully diagnosed by other means.
Hands-On Exercises
Explain, using this chapter's own 15-second CPU spike example, why a dashboard graph showing a "mild" 60% CPU average isn't necessarily telling the whole story.
📄 View solutionExplain why an average response time of 200ms and a p99 of 8 seconds can both be accurate descriptions of the exact same underlying data, and why the p99 figure matters for diagnosing complaints.
📄 View solutionA dashboard shows a completely flat, healthy CPU graph for a server a user says was slow yesterday. Explain what this chapter says to check before concluding CPU wasn't the cause.
📄 View solutionChapter 7 Quick Reference
- Dashboards typically show averaged data — a brief, severe spike can vanish into an unremarkable-looking average
- Switch the aggregation from avg to max on the same panel/window to check for a hidden spike
- Always confirm the time range actually covers the incident before trusting a flat, healthy-looking graph
- Percentiles (p95/p99) beat averages for latency — an average can look fine while a real, painful minority of requests suffer
- A dashboard's genuine strength: correlating multiple panels on the same time axis at once — harder to do one CLI command at a time
- Next chapter: Process-Level Diagnosis: Finding the Specific Culprit