Structured Logging & Log Aggregation

Metrics are done. This chapter moves to the second pillar obs1-1 named — logs — and delivers the real tooling cloud2-4's own correlation-ID material assumed was already in place.

Structured vs. Unstructured Logs

Unstructured logs are free text, meant for a human reading them one line at a time — extracting user_id or duration_ms afterward means writing a regex against the message, fragile and quick to break the moment someone tweaks the wording. Structured logs emit their fields directly, as data — every field is queryable and filterable from the moment it's written, with no parsing guesswork required downstream.

Log Aggregation — Why You Need a Central System

In a system with dozens of services across dozens of instances, logs scattered across that many individual machines are practically unsearchable during a real incident — nobody is going to SSH into thirty containers one at a time while something is on fire. A log aggregation system centralizes logs from every source into one searchable place, which is exactly the capability cloud2-4's own "first five minutes" incident-response technique assumed was already sitting there, ready to use.

Loki vs. the ELK/EFK Stack — Two Different Philosophies

ELK (Elasticsearch, Logstash, Kibana) — or EFK, swapping in Fluentd/Fluent Bit — indexes the full text of every log line in Elasticsearch, enabling powerful free-text search across any word, anywhere, at real storage and compute cost. Loki takes a deliberately different approach: it doesn't index log content at all — only labels, the exact same concept as obs1-2's own Prometheus labels, applied here to logs instead of metrics. The actual log lines are stored compressed, cheaply, and only scanned — not separately indexed — within whichever label-selected stream a query has already narrowed down. This is a deliberate design choice, not a coincidence: Loki is built by the same team behind Grafana specifically to pair with Prometheus's own label-based philosophy, at a fraction of the resource cost full-text indexing requires.

LogQL — Loki's Own Query Language

{job="checkout-service"} is the label selector — cheap, indexed, exactly like a PromQL selector from obs1-4. |= "timeout" is a line filter, applied only within that already-narrowed stream — not indexed, but only scanning a small, pre-selected subset rather than every log line ever written. | json parses each structured line and exposes its fields for further filtering, as in the duration_ms > 5000 example — turning a structured field written once at log time into something directly queryable later.

Correlation IDs — Delivering On cloud2-4's Own Material

cloud2-4 named correlation IDs as the technique for tracing one request across multiple services' own logs, conceptually. Here's the concrete mechanism: a unique ID is generated once, at the edge — typically an API gateway — and passed along on every downstream service call, usually as an HTTP header, then included as a structured field in every log line each service emits while handling that request.

That single query, matching every job, filters down to every log line — across every service that touched this one request — sharing that exact request_id. This is the concrete technique that turns "find every log line related to this one failing request, across the whole system" from a conceptual goal into something you can actually run. obs1-8's traces are the even more powerful version of this same underlying idea — correlating not just log lines, but full per-service timing across the whole request.

Hands-On Exercises