Exercise 3: Why for: 30s Everywhere "To Be Safe" Backfires — Possible Solution ==================================================================== -- What's likely to go wrong -- -- -- 30 seconds is barely enough time to filter out anything -- a single -- slow request, a brief network hiccup, a momentary blip in a metric -- that resolves itself well within a minute would all still be -- enough to satisfy a 30-second for: duration and fire a full alert. -- Applying this "to be safe" instinct uniformly across every single -- rule -- including ones genuinely tracking slower-moving, more -- serious conditions like sustained high error rates or memory -- pressure -- means the team will start receiving pages for -- transient, self-resolving issues that never actually needed a -- human response. Per the chapter's own warn-box, this is exactly the -- mechanism that trains people to start ignoring or dismissing -- alerts reflexively, since they quickly learn that "usually it -- clears itself before I even finish reading the notification" -- -- which defeats the entire purpose of alerting once it happens, -- since a genuinely serious alert now gets the same reflexive, -- distracted response as a false one. -- A better default -- -- -- Rather than one blanket duration "to be safe," the for: value -- should be chosen per rule, based on how quickly that specific -- condition could plausibly resolve on its own versus how long it -- needs to persist before it represents a real, actionable problem. -- The chapter's own HighErrorRate example uses 10m specifically -- because a genuinely serious, sustained error-rate problem is -- expected to still be true 10 minutes later, while a one-off spike -- from a single bad deploy retry or a brief downstream hiccup -- typically isn't. A reasonable general default for most -- non-trivial rules sits somewhere in the 5-15 minute range rather -- than 30 seconds -- long enough to filter out the kind of transient -- blips that resolve themselves, short enough that a genuinely -- ongoing problem still gets caught promptly. WHY THIS WORKS AS AN ANSWER ------------------------------ This traces the specific failure mode of a uniformly short for: duration back to the chapter's own alert-fatigue warning, then proposes a concrete, reasoned default range rather than just asserting "30 seconds is too short" without justification.