Mitigate vs. Fix: Buying Time Safely Without Losing the Evidence

Two of this subject's own technical courses opened with a warning: restarting a struggling process destroys the exact evidence needed to diagnose it. This chapter gives that warning its honest other half — sometimes stopping user pain immediately genuinely is the right call, and it doesn't have to come at the full cost that warning implies.

Two Different Goals, Often Conflated

Mitigating means restoring service quickly — a rollback, a restart, disabling a feature flag, failing over to a backup — without necessarily understanding the actual root cause yet. Fixing means actually understanding and resolving that root cause, so it doesn't recur. These aren't competing goals; they're often correctly sequenced one after the other, mitigate first, fix later — and for a genuinely severe, ongoing incident, that order is usually the right one, not a shortcut.

When Mitigating First Is the Right Call

For a genuinely severe (recall Chapter 3's SEV1/SEV2) ongoing incident, every additional minute has a real cost, and a full root-cause diagnosis can legitimately take longer than users should reasonably have to wait. Stopping the bleeding immediately is usually more valuable than a slower, fully-understood fix delivered at the same speed the incident is still actively hurting people.

The Deliberate Middle Path: Capture Before You Mitigate

The apparent tension between "mitigate fast" and "don't destroy evidence" isn't actually a contradiction — it's a sequencing problem with a small, critical step in between. When even a little time exists before mitigating — seconds to a couple of minutes, often available even during a genuine emergency — capturing a quick snapshot of the current state first preserves the key evidence a full diagnosis will need later, even though the mitigation itself is about to change or wipe out the live state.

This isn't always possible — a truly critical SEV1 might genuinely have zero seconds to spare. But when there is a little time, this single step is what turns "mitigate fast" and "preserve the evidence" from opposing goals into a correctly-ordered sequence.

Don't Forget the Fix Once the Mitigation Works

A real, common failure mode: once the immediate pain stops, there's a strong pull to treat the incident as closed and move on, leaving the actual root cause never properly diagnosed. System Monitoring & Performance Diagnosis's own capstone shows exactly this pattern taken to its natural extreme — weeks of nightly restarts masking a genuine memory leak, never actually fixed because the mitigation kept working well enough that nobody circled back. A mitigation buys time; it doesn't substitute for the fix. A ticket shouldn't close until the root cause is genuinely understood, or a deliberate, documented decision is made not to pursue it further — which is a different thing entirely from simply forgetting.

Working Example: A 30-Second Capture Before a Rollback

A checkout service throws errors for every user — a genuine SEV1. The fastest mitigation is rolling back to the previous deploy. Before doing it, the on-call engineer spends 30 seconds: copying the exact error message and its correlation ID, and confirming via a quick check that no in-flight payment transaction is mid-write, so the rollback won't risk a partial or duplicate charge. Then the rollback happens. Service recovers in under two minutes. The captured error and correlation ID are exactly what let the team properly diagnose and fix the actual bug in the new deploy the next day — the mitigation bought time without becoming a permanent, undiagnosed workaround.

Hands-On Exercises