Sustained vs. Transient Load: When a Spike Isn't a Problem
System Monitoring & Performance Diagnosis
Chapter 9 · Sustained vs. Transient Load: When a Spike Isn't a Problem
Every earlier chapter has been about confirming a genuine problem. This one is the deliberate flip side — recognizing elevated resource usage that's actually normal, bursty, expected behavior, and telling it apart from a real, sustained trend that genuinely needs action. Chapter 1's own baseline theme (an 80% CPU batch server being perfectly normal) gets its fullest treatment here.
Transient Spikes: Normal, Bursty Behavior
Plenty of legitimate causes produce a real, visible spike that isn't a problem at all: application startup (populating caches, JIT warmup), a scheduled batch job, a genuine burst of real traffic (a marketing email going out, a product launch), and — in managed-memory languages — garbage collection pauses, a normal, expected part of how the runtime reclaims memory, not inherently a bug.
Sustained Trends: The Kind That Actually Needs Action
A genuine sustained trend looks different from a transient spike in one specific way: it doesn't fully return to where it started. Real, sustained growth usually means one of two things — organic growth (more real users, more real data, a genuine capacity question) or a resource leak (a bug causing gradual, unbounded consumption that's never actually released, even during quiet periods when load is low).
The "Floor" Technique: Comparing Quiet Periods, Not Peaks
The single most useful technique in this chapter: instead of watching the peaks, watch the low points — the quiet periods, night after night, week after week. A transient spike, no matter how dramatic, returns fully to the same floor each time. A real leak or genuine growth trend leaves a slightly higher floor after every cycle, even if the peaks themselves look similar day to day.
A Genuine Judgment Call: When to Escalate a Trend
Recognizing a real, sustained trend is necessary, but it isn't the same as knowing what to do about it. Whether a confirmed trend needs urgent action, a scheduled fix, or genuine capacity planning depends on context this course can't fully supply on its own — how close the trend is to a hard limit, how fast it's actually climbing, and whether it represents real, wanted growth or a bug that needs fixing. Recognizing the trend is this chapter's job; deciding the response is a separate judgment call, often above what a single support engineer decides alone.
Working Example: The Nightly Restart That Was Never a Fix
A dashboard shows memory climbing to a concerning 85% by the end of every day — but a nightly scheduled restart drops it back to a comfortable-looking 40% each time, and the pattern has repeated, unremarked-on, for weeks. This is worth pausing on directly: Chapter 1 warned that a restart destroys the evidence needed to diagnose a problem. Here's the longer-term version of that same warning — a nightly restart didn't just destroy evidence once, it's been quietly masking a real problem for so long that nobody remembers it's a problem at all.
Applying the floor technique — checking available memory at the exact same point each day, right after the restart, over several weeks — tells a very different story than the daily graph alone:
| Week | Available memory, right after restart |
|---|---|
| 3 weeks ago | 9.2 GB |
| 2 weeks ago | 8.1 GB |
| Last week | 6.9 GB |
| This week | 5.8 GB |
Even the "reset" state is degrading, week over week — a real, slow leak, one that the nightly restart schedule has been tolerating rather than fixing. Left alone, the same trend that's been climbing for weeks eventually reaches the point where a nightly restart is no longer enough to buy back the space, and the actual leak — never diagnosed, only postponed — finally causes a real incident.
Hands-On Exercises
Explain the difference between a transient spike and a sustained trend, using this chapter's own definition of what distinguishes the two.
📄 View solutionExplain the "floor" technique this chapter describes, and why comparing quiet-period low points reveals a real leak that looking only at daily peaks might miss.
📄 View solutionExplain how this chapter's worked example connects back to Chapter 1's own warning about restarts destroying evidence, and why weeks of nightly restarts made the underlying problem harder to notice, not easier.
📄 View solutionChapter 9 Quick Reference
- Transient spikes (startup, batch jobs, real traffic bursts, GC pauses) are normal and resolve fully on their own
- A genuine sustained trend doesn't fully return to where it started — organic growth or a real leak
- The "floor" technique: compare quiet-period low points over time, not peaks — a healthy system returns to the same floor every time; a real problem leaves a slightly higher floor each cycle
- Recognizing a trend is this course's job; deciding the response (capacity planning vs. a bug fix) is a separate, context-dependent judgment call
- A recurring "fix" (like a nightly restart) can mask a real problem for weeks — echoing Chapter 1's own restart-destroys-evidence warning, at a longer timescale
- Next chapter: Capstone: Triaging Three Real Performance Tickets