Alerting with Alertmanager

cloud1-9 named alert fatigue as a real problem back in the Cloud Platforms course, without the tooling to actually fix it. This chapter is where that gets addressed directly — Prometheus's own alert rules, and Alertmanager, the separate component that decides who actually gets told, and how.

Two Separate Jobs — Prometheus Rules vs. Alertmanager

Prometheus itself evaluates alert rules — PromQL expressions that, when true, mark something as firing. That's a deliberately separate concern from Alertmanager, which receives fired alerts and decides who gets notified, through which channel, and on what schedule. Prometheus decides what is wrong; Alertmanager decides who hears about it and how.

Alert Rules — Defining "This Is a Problem"

expr reuses obs1-4's own error-rate query directly. labels attach values used for routing — severity: critical is what a routing rule matches against, not something a human reads. annotations are the reverse — human-readable context, with templating access to the alert's own labels and the value that triggered it, meant to be read by whoever gets paged.

The for: Duration and Alert States

for: 10m means the condition has to stay true continuously for ten minutes before the alert actually fires — a deliberate guard against a single brief blip triggering a page. An alert moves through real, named states: pending (the condition just became true, but for hasn't elapsed yet), firing (the full duration has passed, and it's now been sent to Alertmanager), and resolved (the condition is no longer true — Alertmanager can notify on this too, so a resolved incident doesn't require someone to manually confirm it's over).

Alertmanager — Routing, Grouping, and Silencing

• Routing — a tree of label matchers deciding which receiver (Slack, email, PagerDuty, ...) a given alert goes to, based on its own labels.
• Grouping — multiple alerts sharing similar labels get bundled into one notification, rather than flooding a channel with dozens of separate messages during one widespread incident — directly answering cloud1-9's own alert-fatigue warning.
• Silencing — temporarily mutes alerts matching a given label set, for planned maintenance or a known ongoing issue, without touching the underlying rule itself.

A Routing Tree Example

group_wait is how long Alertmanager waits after the first alert in a new group before sending the initial notification, giving related alerts a chance to arrive and be bundled together. group_interval controls how long to wait before sending an update about new alerts joining an already-notified group. repeat_interval is how long to wait before re-sending a notification for a group that's still firing, unresolved — these three timers together are what actually implement grouping's own noise reduction, not just a conceptual idea.

Avoiding Alert Fatigue — Directly Building On cloud1-9

cloud1-9 named the problem without the tooling; this chapter's routing, grouping, and the for duration together are the concrete fix. Route by real urgency — not everything to the same pager. Group related alerts so one incident produces one notification, not fifty. And before ever setting severity: critical, ask the honest question: if this fires at 3am, does someone genuinely need to get out of bed for it? An alert that doesn't clear that bar belongs at a lower severity, or shouldn't page a human at all.

Hands-On Exercises