Capstone — Building a Full Observability Stack

Observability

Chapter 11 · Capstone — Building a Full Observability Stack

The final chapter. This capstone wires Prometheus, Grafana, Loki, Jaeger, Alertmanager, and the OpenTelemetry Collector together into one real stack, running the exact checkout-service scenario obs1-1 opened this course with — and closes by walking one real incident through it end to end, using all three pillars the way an actual investigation would.

The Architecture — One Pipeline, Three Backends

Four OTel-instrumented services — gateway, auth-service, checkout-service, inventory-service — running in Kubernetes. Each exports telemetry via OTLP to an OpenTelemetry Collector running as a DaemonSet (obs1-10). The Collector fans out to Prometheus (metrics), Loki (logs), and Jaeger (traces). Grafana sits on top, querying all three as data sources (obs1-5). Alertmanager receives fired alerts from Prometheus and routes them (obs1-6). One pipeline, configured once, feeding every backend this course built.

Step 1 — Discovery via ServiceMonitors

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: checkout-stack
spec:
  selector:
    matchLabels:
      part-of: checkout-stack
  endpoints:
    - port: metrics
      interval: 15s

One ServiceMonitor, matching a shared part-of: checkout-stack label across all four services — obs1-10's own real fix for hand-maintaining a static target list as these pods scale and get rescheduled.

Step 2 — Instrumentation via OpenTelemetry

Auto-instrumentation covers incoming/outgoing HTTP calls and database queries across all four services with no code changes. checkout-service adds one manual span for its own business logic — the exact obs1-8/obs1-9 pattern:

with tracer.start_as_current_span("reserve_inventory") as span:
    span.set_attribute("order.item_count", 3)
  

Step 3 — Metrics & Dashboards

Every service exposes http_requests_total (counter) and http_request_duration_seconds (histogram) via auto-instrumentation, plus kube_deployment_status_replicas_unavailable from kube-state-metrics (obs1-10). One Grafana dashboard, built with a $service template variable (obs1-5), covers all four services with three panels: request rate, error rate (obs1-4's own percentage-of-5xx pattern), and p99 latency via histogram_quantile() — one dashboard, not four.

Step 4 — Alerting

groups:
  - name: checkout-stack
    rules:
      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (job) / sum(rate(http_requests_total[5m])) by (job) > 0.05
        for: 10m
        labels:
          severity: critical

obs1-6's own rule, reused directly, now grouped by (job) so each service's error rate is tracked independently. Routed critical → PagerDuty, warning → Slack, grouped by alertname/job so a widespread incident produces one notification, not one per affected instance.

Step 5 — Logs

{"timestamp": "2026-07-13T14:02:11Z", "level": "error", "msg": "inventory reservation timeout", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"}

Structured JSON (obs1-7), with trace_id attached automatically by the OTel SDK (obs1-9) rather than a hand-engineered correlation field. Any log line from any of the four services can be found via one LogQL query filtered on that exact trace_id.

Step 6 — Traces

Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736

gateway (4.1s)                    [root span]
  auth-service (0.1s)
  checkout-service (3.9s)
    reserve_inventory (3.85s)
      inventory-service (3.8s)
        db-query (3.75s)

obs1-1's own opening example, now a real trace flowing through the actual pipeline built across this entire course — 3.8 of 4.1 seconds spent in inventory-service, visible directly in Jaeger's own waterfall view.

A Real Incident, Walked End to End

Alertmanager fires HighErrorRate for checkout-service, routed to Slack (severity: warning at first, since error rate hasn't yet crossed into timeouts becoming outright 5xx failures).
Grafana's dashboard, filtered to $service=checkout-service, shows the p99 latency spike and pinpoints the time window.
Explore mode against Loki, filtered to that time window and service, surfaces the "inventory reservation timeout" log line and its trace_id.
Jaeger, queried by that exact trace ID, shows the full tree above — inventory-service's own db-query span is where the time actually went, not checkout-service itself.

Metrics said something was wrong; logs said what; traces said exactly where — obs1-1's own three-question framing, now answered by a real, working pipeline rather than a conceptual walkthrough.

Piece	Chapter
ServiceMonitor discovery	obs1-10
OTel instrumentation & Collector pipeline	obs1-8, obs1-9
PromQL error-rate/latency queries	obs1-4
Grafana dashboard with template variable	obs1-5
Alertmanager routing & grouping	obs1-6
Structured logs, LogQL, correlation	obs1-7
Trace tree, span attributes	obs1-8
Metrics vs. logs vs. traces framing	obs1-1, obs1-2

Still out of scope, honestly

This capstone doesn't cover long-term, cross-cluster storage (Thanos/Cortex/Mimir, explicitly named out of scope back in obs1-3), production-grade high availability for the observability stack itself (a single Prometheus/Loki/Jaeger instance each, not a replicated deployment), securing the stack's own endpoints (dashboards and metrics endpoints need their own access control, a separate concern from anything this course covered), or a real tail-based sampling implementation (obs1-8 discussed the strategy, not the deployment). None of these are oversights — they're genuine next steps beyond what an 11-chapter course can reasonably claim to have finished.

Hands-On Exercises

Exercise 1

Write the LogQL query that would find the exact log line shown in Step 5, filtered to the checkout-stack's logs and the specific trace_id shown in Step 6.

📄 View solution

Exercise 2

Explain why the incident walkthrough moves from Alertmanager to Grafana to Loki to Jaeger in that specific order, rather than starting directly with Jaeger.

📄 View solution

Exercise 3

A colleague asks why this capstone doesn't include a production-HA setup for Prometheus/Loki/Jaeger themselves. Write a short, honest answer using this chapter's own scope note.

📄 View solution

Chapter 11 Quick Reference — Observability Complete

One OTel Collector pipeline feeds Prometheus, Loki, and Jaeger from the same instrumented services — obs1-9's own unification, made real
ServiceMonitors handle discovery; Grafana with template variables handles visualization; Alertmanager handles routing/grouping
trace_id ties logs and traces together automatically; metrics point at the right time window to search within
The incident walkthrough is obs1-1's own three-pillar framing (something's wrong / what happened / where) answered by a real, working pipeline
Honest scope note: no long-term storage, no HA for the stack itself, no stack-endpoint security, no tail-based sampling deployment — genuine next steps, not oversights
The full Observability course — 11 chapters — is now complete.