Reading Logs & Metrics Under Pressure

Connectivity (Chapter 2) and IAM (Chapter 3) covered specific problem categories. This chapter is about a skill that cuts across all of them: reading monitoring data effectively while an incident is actively happening, when time pressure genuinely changes how the investigation should proceed. It builds directly on cloud1-8's metrics-then-logs workflow — the theory covered there, practiced properly here.

Restating the Foundation — Cloud1-8's Metrics-Then-Logs Workflow

Recall the sequence: metrics narrow down when and roughly where; logs explain what actually happened. This chapter is about doing that efficiently and correctly under real time pressure, not just knowing the theory.

The First 5 Minutes of an Incident

Resist the urge to start reading logs immediately. First, establish the metrics-level picture: what changed, when did it start, and how widespread is it — one instance or all of them, one region or all regions (cloud1-1's regions/AZs). This initial triage determines the scope of the investigation before diving into detail. Skipping straight to logs risks getting lost in fine-grained detail for a problem that's actually much narrower — or much broader — than initially assumed.

Correlating Logs Across Multiple Services

A single user-facing failure often spans multiple services — a load balancer, web servers, a database (cloud2-1's architecture-diagram material) — each potentially logging to a different aggregation location (cloud1-8). The practical technique: if the application logs a shared correlation ID (a request or trace ID) with each request, use it to filter every service's logs down to just the entries relevant to one specific failing request, rather than manually scanning unrelated traffic across each source separately. Without a correlation ID, fall back to narrowing by time window plus the likely source IP or resource identifier as the next-best approach.

Effective Log Query Techniques Under Time Pressure

• Start broad, then narrow — time range plus service first, then a specific error pattern or status code — rather than trying to write the "perfect" query immediately.
• Absence can be diagnostic too — a service that should log "request received" but doesn't suggests the request never actually reached it at all, a genuinely useful reframing that's easy to overlook.
• Keep a small personal library of go-to queries for common scenarios, rather than rebuilding syntax from memory during a live incident — a genuinely practical time-saving habit.

Distinguishing Correlation From Causation in Metrics

A real, common mistake under pressure: two things happening around the same time doesn't mean one caused the other. CPU spiking exactly when errors started could be the errors causing retries and extra load — a symptom, not the cause. The useful discipline: look at what changed first in the timeline, not just what looks abnormal right now, and stay willing to revise an initial hypothesis if later evidence doesn't fit it.

When Alert Fatigue Becomes a Real Problem Mid-Incident

Revisiting cloud1-8's alert fatigue material specifically during an active incident: a flood of secondary, downstream alerts — services failing because of the root cause, not the root cause itself — can bury the one alert that actually points at the real problem.

Documenting What You Found, As You Go

A genuinely practical habit: jot down timestamps, findings, and ruled-out theories as the investigation happens, not reconstructed afterward from memory. This is directly useful for Chapter 6's incident response and postmortem material, and it also simply prevents re-checking the same thing twice during a long, stressful investigation.

Hands-On Exercises