Common Failure Modes & Root Cause Analysis

Chapter 4 covered investigation technique. This chapter catalogs the actual failure patterns a support engineer will run into repeatedly — recognizing a familiar pattern is genuinely faster than diagnosing every incident entirely from first principles.

Why a Catalog of Common Failures Is Worth Having

This isn't an exhaustive list — it's the failure modes common enough to be worth having pre-loaded, directly echoing Chapter 4's "keep a go-to query library" habit: this is the analogous library, but for recognizing failure patterns rather than writing queries.

Instance Failures

• Hardware failure — rare but real; the underlying host has an issue, sometimes surfaced in advance via provider-initiated retirement notices.
• OS-level crashes/kernel panics — often visible in system logs beforehand, if caught in time.
• Out-of-memory (OOM) kills — the OS killing a process to free memory, a genuinely common and diagnosable pattern via specific log signatures.
• Health-check-triggered replacement (cloud1-3/cloud1-5) — the instance may look like it failed when the health check itself was actually misconfigured (checking the wrong port or path) — a real, common false positive worth naming explicitly.

Storage Throttling

Revisiting cloud1-4: managed storage volumes typically have IOPS/throughput limits tied to volume size or type. A workload exceeding those limits gets throttled, not failed outright — performance degrades rather than producing an obvious error, which is genuinely confusing since nothing "broke" in the usual sense. The symptom pattern: increased latency correlating with high disk I/O metrics specifically, not CPU. The fix is usually resizing the volume or moving to a higher-performance storage tier — not debugging application code.

Database Connection Exhaustion

Revisiting cloud1-7 as a diagnosable pattern: "too many connections" errors appearing suddenly under load, or right after a deployment that introduced a connection leak (connections not properly closed or returned to the pool). To confirm: check the database's current connection count against its configured maximum — a direct diagnostic. A genuinely common root cause worth naming specifically: a recent code deployment changing connection pooling behavior — directly tying back to Chapter 4's "what changed and when" discipline.

Auto-Scaling Misconfigurations

Revisiting cloud1-3's auto-scaling masking material, now cataloged as specific, diagnosable patterns:

• Scaling too slowly — an overly conservative trigger or cooldown period means new instances come online after the spike has already caused user-visible failures.
• Scaling on the wrong metric — scaling on CPU when the real bottleneck is memory or database connections, so adding instances never actually relieves the real constraint.
• Hitting the max instance limit — cloud1-3's own "eventually you hit the max" gotcha, now a directly checkable diagnostic: is the current instance count sitting right at its configured maximum?

A Root Cause Analysis Framework

The 5 Whys technique: repeatedly ask "why did that happen" past the first, surface-level answer, to avoid stopping at a symptom rather than the actual root cause.

That last answer is a genuine root cause worth fixing — versus stopping at "the database was slow" and just restarting it, which fixes nothing long-term.

Distinguishing a True Root Cause From a Contributing Factor

A genuinely useful nuance: sometimes there are multiple contributing factors, not one single clean root cause. In the example above, both the connection leak (a code bug) and an undersized connection pool (a config choice) could plausibly have contributed — fixing only one might reduce frequency without eliminating the risk entirely. Worth being honest about this in a postmortem rather than declaring an artificially clean single cause.

Hands-On Exercises