Distributed Tracing

obs1-7 closed with a promise: traces are the more powerful version of the same correlation idea logs only partially deliver. This chapter is where that becomes concrete — the third pillar, and genuinely new material nothing else on this site has covered before now.

What a Trace Actually Is

A trace represents one request's full journey across every service it touches. It's built from spans — each span represents one unit of work: one service handling the request, or one specific operation inside it, like a database query. A span has a name, a start time, a duration, and a set of key-value attributes, similar in spirit to a metric's own labels.

Spans — The Building Block

Every span has a unique span ID, and — except the very first, root span — a parent span ID, forming a tree that reconstructs the exact call graph a request actually took. Every span belonging to the same request shares one trace ID, the thread tying the whole tree together.

This is obs1-1's own checkout example, made literal. "3.8 of the 4 seconds spent waiting on inventory" isn't a conclusion someone had to piece together from separate logs anymore — it's a visible span in this exact tree, with its own duration sitting right there.

Trace Context Propagation

For spans created in separate services to link into one trace, the trace ID — and the current span ID, to establish the parent-child relationship — has to travel along with every downstream call, usually as an HTTP header. This is mechanically the same idea as obs1-7's own correlation ID, formalized into a real, standardized format: the W3C Trace Context traceparent header, rather than every team inventing its own ad-hoc header.

Four dash-separated parts: version, trace ID, the parent span's own ID, and trace flags. Every hop that receives this header, creates its own span as a child of that parent ID, and forwards an updated version downstream — the exact mechanism that turns separate services' separate spans into one connected tree.

OpenTelemetry — A First Look at Instrumenting Your Code

OpenTelemetry (OTel) is the vendor-neutral standard for generating this telemetry — an SDK application code calls directly to create spans, which are then exported to a tracing backend. obs1-9 covers it properly; here's the shape of it:

Many frameworks and libraries support auto-instrumentation — OTel can automatically create spans for common operations (incoming HTTP requests, outgoing calls, database queries) with no manual code changes at all, while manual spans like the one above cover anything genuinely business-specific a generic instrumentation library couldn't know to track on its own.

Jaeger — Storing and Visualizing Traces

Jaeger receives the spans instrumented services export, stores them, and provides a UI to search traces — by service, operation, duration, tags — and to visualize any single trace as a waterfall/timeline view, exactly like the tree above but interactive. It's the tracing pillar's own equivalent of what Grafana is for metrics: a query-and-visualization layer sitting on top of the actual data. And as obs1-5 already noted, Grafana itself can visualize Jaeger (or Tempo, its close relative) traces directly as a data source, right alongside metrics and logs in the same dashboard.

Hands-On Exercises