Exercise 2: Why "Configured and Firing" Notifications Still Failed This Organization — Possible Solution ==================================================================== WHAT "CONFIGURED AND FIRING" ACTUALLY MEANT HERE ------------------------------ Per this chapter, "failure notifications were configured and had been firing the entire time - but to an email inbox that was decommissioned during a recent email system migration." The notification system itself was doing exactly what it was built to do: generating an alert every time the job failed. The failure wasn't in the alerting mechanism - it was in where those alerts were going. WHY A TECHNICALLY WORKING ALERT ISN'T THE SAME AS A USEFUL ONE ------------------------------ An alert's entire value depends on a person actually receiving and acting on it. An alert firing correctly into an inbox nobody reads produces the exact same real-world outcome as no alert firing at all - nobody finds out. Per this chapter's own warn-box, "an alert configured to go somewhere nobody monitors is functionally identical to having no alert at all." The mechanism succeeding and the actual goal (someone finding out) succeeding are two different things, and only the second one actually matters. WHY THE EMAIL MIGRATION SPECIFICALLY CAUSED THIS ------------------------------ The alerts were pointed at a specific inbox that existed and was presumably monitored when the backup job was first configured. The migration changed what existed on the receiving end without anyone updating the alert's own destination - the sending side kept working exactly as before, while the receiving side silently stopped existing. Neither side "broke" in a way that would trigger its own error; the mismatch between them is what caused the silent failure. WHAT SPECIFICALLY CLOSED THE GAP ------------------------------ The gap was closed by this chapter's own recommended practice: periodically confirming that alerts actually reach a real, currently- watched inbox, rather than only checking the alerting configuration when a ticket like this one forces the question. Per this chapter, "periodically confirming that failure notifications actually reach a real, watched inbox is its own separate check" from confirming the alert fires at all. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains precisely what was and wasn't working (firing correctly, but to a dead destination), explains why a technically functioning alert with no actual recipient is worthless, and identifies the specific ongoing check (periodically confirming alerts still reach a live inbox) that would have caught this before three weeks passed.