Capstone — Running One Incident Start to Finish
Incident Response & Ticketing Workflows
Chapter 10 · Capstone — Running One Incident Start to Finish
Nine chapters built the pieces — classification, a good ticket, a live timeline, mitigating safely, escalating well, communicating honestly, and reviewing blamelessly. Unlike this subject's other four courses, this capstone doesn't split into three separate tickets — it runs one complete, fresh incident through every one of those stages in sequence, since a process course benefits more from one coherent story than three disconnected ones.
The Incident: Search Returning Zero Results
Starting around 10:00 UTC, search begins returning no results for a growing number of previously-working queries. Browsing and checkout are unaffected.
Ticket filed at 10:14 UTC: "Search returning no results for common terms since around 10am." Steps to reproduce: search for 'wireless headphones' on the site. Expected: results shown. Actual: empty results, no error visible to the user. Scope: appears to affect only some search terms, not all. Timestamp: onset ~10:00 UTC. No correlation ID yet — no visible error to capture one from, a genuine limit worth noting rather than inventing one that doesn't exist.
Scoping: many users (a dozen similar reports within 15 minutes), one specific feature (search only), sudden onset, correlating with a deploy at 09:55 UTC. Severity: SEV2 — a major feature broken for a significant subset, not a full outage. Impact/Urgency: high impact (search drives significant traffic) and rising urgency (actively worsening as more terms are affected) — act now, per Chapter 3's own matrix.
Notice the 10:24 entry stays in the record — a reasonable theory, checked and ruled out, exactly as Chapter 5 requires.
Decision: roll back v3.2 immediately — SEV2, actively worsening. Before rolling back, a 90-second capture: the exact query-format diff between v3.1 and v3.2, three example failing search terms with timestamps, and the index backend's own version number. A quick harm check: search is read-only, no in-flight writes at risk — the rollback itself is safe. Rolled back at 10:44 UTC.
Technical escalation, sent alongside the rollback decision: "Escalating to Search Infra — SEV2, v3.2's query-format change is incompatible with the current index backend contract (diff attached). Rolling back v3.2 as immediate mitigation, effective ~10:44 UTC. Need: a target timeline for updating the index backend to support the new format, so a compatible v3.2 re-deploy can be planned properly." No management escalation was needed this time — resolved within the hour, no business decision required. Not every incident needs both kinds, exactly as Chapter 7 notes.
Chapter Attribution
| Stage | Source chapter |
|---|---|
| Ticket intake with the core actionable elements | Chapter 4 |
| Severity classification (SEV2) | Chapters 2–3 |
| Priority via the Impact/Urgency matrix | Chapter 3 |
| Live timeline, including a ruled-out theory kept in the record | Chapter 5 |
| Capture-before-mitigate, plus a harm check before rolling back | Chapter 6 |
| A technical escalation with evidence and a clear ask; recognizing no management escalation was needed | Chapter 7 |
| A consistent, honest update cadence and an explicit all-clear | Chapter 8 |
| A blameless review with a real root cause and dated, owned action items | Chapter 9 |
| The framing distinguishing this course's own scope from the other four Technical Support courses | Chapter 1 |
Honest Scope Note
- No specific ticketing-tool tutorials (Jira, PagerDuty, and similar) — the concepts transfer directly, but tool interfaces vary too much to cover here
- No formal ITIL-certification-level process depth — this course teaches the practical core, not a certification curriculum
- No legal/compliance-specific incident reporting requirements (e.g. breach-notification timelines) — a genuinely separate, specialized topic
- No crisis PR or media handling for major public-facing incidents — a distinct discipline of its own
- No on-call rotation design or staffing methodology — this course assumes a rotation exists, not how to build one
Hands-On Exercises
Explain why the 10:24 timeline entry (the cache theory, ruled out) stayed in the capstone's record instead of being removed once the real cause was found.
📄 View solutionExplain why no management escalation was needed for this incident, even though it was a genuine SEV2, and what this shows about Chapter 7's own two-audience distinction.
📄 View solutionExplain why the capstone's root cause is "a missing cross-service compatibility check," not "the deploy broke search," and why that distinction changed what the action items actually addressed.
📄 View solutionChapter 10 Quick Reference — Course Complete
- One incident, every stage: classification → ticket → timeline → mitigate-vs-fix → escalation → communication → review
- A ruled-out cache theory stayed in the timeline; the real cause (an incompatible deploy) was found 24 minutes after the ticket landed
- A safe, evidence-preserving rollback resolved the incident in 44 minutes; no management escalation was actually needed
- The review's root cause was a missing CI check, not "a bad deploy" — producing action items that prevent a recurrence, not just today's incident
- This closes Incident Response & Ticketing Workflows, 10/10 chapters — the fifth complete course under the Technical Support subject, alongside Logging & Log Analysis, Network Troubleshooting, System Monitoring & Performance Diagnosis, and Web & Application Troubleshooting