Capstone — Diagnosing a Real Multi-Service Outage

One full incident, worked end to end, combining every technique from this course's nine prior chapters — and, throughout, the Course 1 material each of those techniques was originally built on.

The Scenario

TechShop, a mid-size e-commerce company, launches a major flash sale. Twenty minutes in, checkout starts failing intermittently for a growing share of customers.

Step 1 — Detect & Triage

An error-rate alarm fires on the checkout service (cloud1-8's alerting). Severity triage (Chapter 6) is immediate: checkout is directly revenue-generating, a large and growing share of customers are affected, there's no workaround, and it's happening during a promotional event — this is a Sev1 without hesitation. An incident commander is declared.

Step 2 — Scope Before Diving Into Logs

Following Chapter 4's first-five-minutes discipline: what changed, when did it start, how widespread is it. The picture that emerges — errors are intermittent, not total, and started almost exactly when flash-sale traffic began climbing. This intermittent pattern is the same diagnostic signal cloud2-1's own exercise walked through: a symptom pointing at something failing under load, not a total, clean outage.

Step 3 — Diagnosing Connectivity? Ruled Out Quickly

Applying Chapter 2's flowchart briefly: security groups, NACLs, and route tables are all checked and confirmed unchanged recently — no networking deployment in the relevant window. This step returns clean fast, which is itself useful information: it's genuinely fine, and expected, for a flowchart step to rule something out quickly rather than assuming every incident must be a networking problem.

Step 4 — Correlating Logs Across Services

Using Chapter 4's correlation-ID technique: request IDs are traced from the load balancer, through the application servers, to the database layer. The pattern that emerges — every failing request shows a connection timeout from the application server to the database, specifically.

Step 5 — Recognizing the Failure Pattern

This matches Chapter 5's catalog directly: database connection exhaustion. Confirmed by checking the database's current connection count against its configured maximum — sitting at 100%. Applying "what changed": a deployment two days earlier changed the application's connection pooling library. Normal, pre-sale traffic never came close to the new pool's limit, so the change went unnoticed until flash-sale traffic pushed it over.

Step 6 — Applying the 5 Whys

The genuine root cause: a missing load-testing step in the deployment process — not simply "the pool was too small," which is only a symptom of that deeper gap.

Step 7 — Immediate Mitigation vs. Long-Term Fix

Per Chapter 5's own distinction: the immediate mitigation is increasing the connection pool size and restarting affected instances to clear stuck connections, restoring service quickly. The long-term fix — adding load-testing to the deployment process — is the actual root cause from Step 6. Both matter; stopping at the mitigation alone, per Chapter 5's warn-box, leaves the underlying gap ready to cause the same failure again.

Step 8 — Communicating Throughout

Per Chapter 6: an early "we're investigating" update goes out within minutes of the Sev1 declaration. Regular updates follow on a set cadence. No public speculation about cause happens until Step 5-6 actually confirm it. A clear resolution update follows once the mitigation is applied.

Step 9 — A Related Cost Question Surfaces

During the investigation, someone notices the auto-scaling group scaled out significantly. Is this legitimate cost from genuine load, or cloud1-3's own auto-scaling-masking-a-root-cause gotcha? Here, it's honestly both — the scaling was a reasonable reaction to real load, but it was also masking the connection pool issue underneath the whole time, exactly the pattern cloud1-3 warned about. Per cloud1-9's idle-compute material, the extra instances should be scaled back down once the real fix is in place, not left running unnecessarily.

Step 10 — Was This Us, or the Provider?

Applying Chapter 1's shared responsibility model, checked early per Chapter 8: the provider's account-specific health dashboard showed nothing. This was entirely a customer-side application and configuration issue — not a provider outage — closing off the provider-support escalation path from Chapter 8 as unnecessary here.

The Postmortem

A blameless postmortem (Chapter 6) is written: timeline, the 5 Whys root cause chain from Step 6, and an honest note on contributing factors (Chapter 5) — both the small default pool size and the missing load-testing process contributed, not one clean single cause. Action items with owners:

1. Immediately fix the connection pool size — done, during the incident.
2. Add load-testing to the deployment pipeline — owned by the platform team, with a due date.
3. Add a dedicated connection-count alert (cloud1-8), so this shows up as its own early signal next time, rather than only manifesting as checkout errors after the fact.

What This Capstone Demonstrated

And underneath every one of those: Course 1's own material — cloud1-1's shared responsibility, cloud1-3's auto-scaling and connection concepts, cloud1-7's connection management, cloud1-8's monitoring, cloud1-9's cost management — the foundation this entire course was built to turn into practiced, applied skill.

Hands-On Exercises