cloud1-8 introduced the vocabulary — metrics, logs, traces, automatic vs. configured monitoring, alert fatigue. cloud2-4 built real incident-response discipline on top of that vocabulary — the first-five-minutes triage habit, cross-service correlation IDs, "find the earliest alert, not the loudest." Both stayed conceptual and tool-agnostic, on purpose. This course goes deep specifically on the real tooling underneath — Prometheus, Grafana, Loki, OpenTelemetry, Jaeger — the systems that actually produce the metrics, logs, and traces those two chapters assumed were already there.

Monitoring vs. Observability — A Real Distinction

Monitoring watches for failure modes you already anticipated — a dashboard for CPU usage, an alert for disk space below 10%, a check for "is the process running." It answers questions decided in advance. Observability is different: the ability to ask a new question about a system's internal state, using only its external outputs, without shipping new code to answer it. cloud2-4's own "first five minutes" already implied this — during a genuinely novel incident, you usually don't know what to look for yet. Monitoring gives you the dashboards someone thought to build in advance; observability gives you the raw material to explore a question nobody anticipated.

The Three Pillars

Three distinct categories of telemetry, each answering a different kind of question:

• Metrics — numeric measurements over time (request rate, CPU%, P99 latency). Cheap to store, efficient to aggregate, ideal for trends and alerting — but they summarize; a metric can't tell you what happened to one specific failed request.
• Logs — discrete, timestamped event records, as rich and arbitrary as the code that emits them. Detailed, but expensive to store and query at real scale, and hard to correlate across services without deliberate effort — exactly the correlation-ID discipline cloud2-4 already named.
• Traces — the path one specific request takes across every service it touches, with timing for each hop. The pillar built specifically to answer "where, in this whole chain, did the time actually go?"

A Concrete Walkthrough — One Request, Three Views

A checkout request is slow. Each pillar answers a different piece of the same incident:

• Metrics say: P99 checkout latency has climbed from 200ms to 4s over the last ten minutes — a trend, aggregated across every request, telling you something is wrong and roughly when it started.
• Logs say: the payment service logged three timeout errors and a retry in the last minute — detail, but only for the services that happened to log something, and only if you already know which service to look at.
• Traces say: for this specific slow request, 3.8 of the 4 seconds were spent waiting on a single downstream call to the inventory service — the one view that actually pinpoints where in the chain the time went, rather than just confirming that some slowdown exists.

No single pillar answers the whole question alone. Metrics tell you something is wrong; logs tell you what a given service saw; traces tell you where, across the whole request's path, the problem actually lives.

Hands-On Exercises

obs1-1 named metrics as the pillar that tells you something is wrong, cheaply, in aggregate. This chapter goes underneath that claim: exactly how Prometheus — the tool this course builds around for the metrics pillar — actually represents a metric. Everything in Ch.3 (scraping) and Ch.4 (PromQL) builds directly on the data model established here.

A Metric Is a Time Series

A Prometheus metric isn't one number — it's a named stream of (timestamp, value) pairs collected over time, identified by a metric name plus a set of labels, key-value pairs attached to that specific series.

http_requests_total is the metric name; the three labels together identify exactly which time series this particular value belongs to. This isn't one running total for the whole application — it's one time series among potentially many under the same metric name.

Labels — The Dimension That Makes Aggregation Possible

Every unique combination of label values creates a genuinely separate time series:

Three distinct time series, one metric name. This is exactly what makes Chapter 4's PromQL genuinely useful — labels are the dimension you filter and aggregate along, letting a single instrumented metric answer "what's my GET rate," "what's my 500 rate," and "what's my total request rate across every method and status combined" from the same underlying data.

The Four Metric Types

Prometheus defines exactly four kinds of metric, each with a different shape of value and a different intended use:

• Counter — only ever increases (or resets to zero on restart). Right for "total requests served," "total errors," anything that's a running cumulative count. A raw counter value is rarely useful by itself; Chapter 4's rate() turns it into a meaningful per-second figure.
• Gauge — a value that can go up or down freely. Right for "current memory usage," "active connections," "queue depth right now."
• Histogram — sorts observations (like request durations) into configurable buckets, alongside a running sum and count. Quantiles are computed later, at query time, from the bucket counts.
• Summary — similar intent to a histogram, but quantiles are calculated client-side, inside the instrumented application itself, before the metric is ever exposed.

Histogram vs. Summary — A Real Operational Tradeoff

These two look similar but behave very differently once there's more than one instance of a service running. A histogram's raw bucket counts from many different pods can simply be summed together — Chapter 4's histogram_quantile() then computes one fleet-wide p99 from the combined buckets. A summary's quantile, by contrast, is already computed inside one specific instance before it's ever exposed — there is no meaningful way to average two different instances' own p99 values together and get a real fleet-wide p99. For anything that might ever need aggregating across replicas — which, in practice, is nearly everything in a real production system — histograms are the safer default.

Hands-On Exercises

obs1-2 established what a metric looks like once it exists. This chapter covers how it actually gets from an application into Prometheus in the first place — the architectural choice that shapes nearly everything else about how the system is operated.

The Pull Model — Prometheus Comes to You

Prometheus's core design decision: it scrapes (pulls) metrics by periodically sending an HTTP GET to a /metrics endpoint each target exposes, in the exposition format from obs1-2. Applications don't send their metrics anywhere — they just expose them on an endpoint and wait to be asked, on whatever schedule Prometheus's own configuration decides.

Why Pull? — Real Advantages

• Centralized control — scrape frequency and target lists live in one place (Prometheus's own config), not scattered across every application's own settings.
• A failed scrape is itself a signal — if Prometheus can't reach a target, that's meaningful information ("this target is unreachable") distinct from "this target reported zero" — something a push-based system can't easily distinguish from silence.
• Simpler service discovery — Prometheus decides who to scrape and finds them; individual services never need to know Prometheus's own address.
• Trivially testable locally — curling a /metrics endpoint shows exactly what Prometheus would see, with no special tooling required.

When Pull Doesn't Fit — Pushgateway

A short-lived batch job can finish and exit before any scheduled scrape would ever reach it — the pull model has nothing to pull from once the process is gone. The Pushgateway is the sanctioned exception: a batch job pushes its final metrics to the Pushgateway once, right before exiting, and Prometheus then scrapes the Pushgateway itself, like any other ordinary target.

Exporters — Getting Metrics Out of Things That Don't Speak Prometheus

Most systems — MySQL, Redis, the Linux kernel itself, older applications — don't natively expose a Prometheus-format /metrics endpoint. An exporter is a small adapter process that queries the underlying system in whatever native way it supports, then re-exposes that data as an ordinary Prometheus-format endpoint for scraping. This is exactly the missing piece cloud1-8's own terminology-level coverage never got into.

• node_exporter — host-level metrics: CPU, memory, disk, network, straight from the OS
• mysqld_exporter / redis_exporter — connect to the database using its own native protocol, re-expose the results as Prometheus metrics
• blackbox_exporter — actively probes external endpoints (HTTP, TCP, ICMP) and reports reachability/latency as metrics

Configuring Scrape Targets — prometheus.yml

Each job_name groups a set of targets sharing the same scrape configuration. Every scraped sample is automatically tagged with a job label (and an instance label identifying which specific target), giving every metric a built-in "which service, which instance" dimension for free.

Service Discovery — Beyond Hardcoded Targets

A static, hand-maintained target list breaks the moment instances scale up, scale down, or get replaced — exactly what happens continuously in Kubernetes or a cloud autoscaling group. Prometheus supports service discovery mechanisms — kubernetes_sd_config, ec2_sd_config, consul_sd_config, among others — that automatically discover the current set of scrape targets directly from the underlying platform, rather than requiring anyone to keep a list in sync by hand. obs1-10 covers Kubernetes' own version of this in depth, via ServiceMonitors and the Prometheus Operator.

Retention — Local Storage Isn't Forever

Prometheus stores scraped samples on local disk in its own time-series database, with a configurable retention window — 15 days is a common default. This is a deliberate scoping decision, not an oversight: Prometheus is built for reasonably recent operational data, not as a long-term archive. For genuinely long-term retention or a unified view across multiple Prometheus instances, remote-writing to a dedicated long-term storage backend (Thanos, Cortex, Mimir) is the standard real-world answer — worth naming honestly here as beyond this course's own scope, a deliberate line drawn rather than an oversight.

Hands-On Exercises

obs1-2 left a promise on the table: a raw counter value is rarely useful by itself. This chapter delivers on it — PromQL, the query language that turns Ch.2's raw time series into the numbers a dashboard or an alert actually cares about.

Instant Vectors vs. Range Vectors

http_requests_total alone is an instant vector — the current value of every matching time series, at one point in time. http_requests_total[5m] is a range vector — every value each matching series took over the trailing five minutes. Range vectors can't be graphed directly; they exist specifically as input to functions like rate() that need a window of history to compute something meaningful.

rate() and irate() — Turning a Counter Into Something Useful

rate() computes the per-second average rate of increase over the given window — exactly the transformation Ch.2 promised a raw counter needed. It also automatically detects and compensates for counter resets (a process restart dropping the counter back to zero), a genuinely important detail: without that handling, a restart would otherwise show up as a nonsensical negative rate. irate() computes an instantaneous rate using only the last two data points in the range, more responsive to sudden spikes but noisier — rate() is the safer default for dashboards and alerting; irate() suits fast-moving, high-resolution graphs where responsiveness matters more than smoothness.

Aggregation Operators — Collapsing Across Labels

sum(), avg(), min(), max(), and count() combine values across every matching series. The by (...) clause keeps specific label dimensions in the result, collapsing everything else; without (...) does the reverse — drop these labels, keep the rest. Choosing the right dimension to keep is exactly what turns obs1-2's own per-instance, per-status, per-method time series into the one useful number a dashboard panel actually wants.

histogram_quantile() — Delivering on Chapter 2's Own Promise

obs1-2 named histograms as aggregatable specifically because their bucket counts can be summed across instances. This is that promise made concrete: rate() turns each bucket's cumulative count into a per-second rate, sum(...) by (le) combines those rates across every instance while explicitly preserving the bucket-boundary label, and histogram_quantile() then computes an approximate p99 from the combined buckets.

Common Query Patterns

=~ matches a label against a regular expression — "5.." catches every 5xx status code in one pattern. increase() is rate()'s own close relative: instead of a per-second rate, it reports the total increase across the whole window, correctly handling counter resets the same way rate() does — the right choice for "how many of X happened," rather than "how fast is X happening."

Hands-On Exercises

obs1-4 gave you PromQL. This chapter gives you somewhere to put it — Grafana, the visualization layer this course pairs with Prometheus. And in keeping with obs1-1's own opening warn-box, this chapter is honest about what a dashboard, no matter how well-built, actually is and isn't.

What Grafana Actually Is — A Visualization Layer, Not a Data Store

Grafana itself stores no metrics. It's a query-and-render frontend that connects to one or more data sources — Prometheus here, but also Loki for logs (obs1-7) and Jaeger/Tempo for traces (obs1-8) — and turns their query results into panels. This separation matters in practice: losing Grafana loses no actual data, since everything it displays lives in the underlying data source, not in Grafana itself. It also means one Grafana dashboard can show metrics, logs, and traces side by side, sourced from entirely different systems.

Data Sources — Connecting Grafana to Prometheus

A data source is configured once — a name, a type (Prometheus), and the URL Grafana should query. Once that connection exists, every panel in every dashboard can query it directly using ordinary PromQL, exactly as written in obs1-4.

Panels — The Basic Building Block

A panel is one visualization, backed by one or more queries. Common types: a time series graph, a single stat/number, a gauge, a table, and a heatmap — genuinely useful specifically for visualizing a histogram's own bucket distribution over time.

That's the exact error-rate query from obs1-4, dropped straight into a panel — a panel is nothing more than a chosen visualization wrapped around a PromQL query you already know how to write.

Template Variables — One Dashboard, Many Contexts

$service is a template variable — a dropdown at the top of the dashboard, typically populated by a label_values() query against Prometheus itself, that gets substituted into every panel's query at render time. One dashboard, built once, can then serve every service sharing that shape of metric — rather than hand-building a near-identical dashboard per service.

Dashboards as Code — Provisioning

Building a dashboard by clicking through Grafana's own UI works, but doesn't scale and isn't reviewable the way real code is. Grafana dashboards can instead be defined as JSON, checked into version control alongside application code — the same review-and-history discipline git1-3 already established for everything else. Provisioning is what makes this practical: Grafana can automatically load dashboard JSON and data source configuration from files at startup, rather than requiring anyone to manually recreate them through the UI — a light echo of k8s2-9's own GitOps pattern, applied here to dashboards instead of Kubernetes manifests.

An Honest Note — Dashboards Are Still Monitoring, Not Observability

obs1-1's own warn-box applies here directly: a dashboard, however well-designed, only ever answers the questions its panels were built to answer in advance. Grafana's Explore mode is its own genuine answer to that limit — an interface for running ad-hoc PromQL queries against a data source without needing a saved panel first, letting you follow a brand-new question on the spot during an incident rather than being limited to whatever panels already exist.

Hands-On Exercises

cloud1-9 named alert fatigue as a real problem back in the Cloud Platforms course, without the tooling to actually fix it. This chapter is where that gets addressed directly — Prometheus's own alert rules, and Alertmanager, the separate component that decides who actually gets told, and how.

Two Separate Jobs — Prometheus Rules vs. Alertmanager

Prometheus itself evaluates alert rules — PromQL expressions that, when true, mark something as firing. That's a deliberately separate concern from Alertmanager, which receives fired alerts and decides who gets notified, through which channel, and on what schedule. Prometheus decides what is wrong; Alertmanager decides who hears about it and how.

Alert Rules — Defining "This Is a Problem"

expr reuses obs1-4's own error-rate query directly. labels attach values used for routing — severity: critical is what a routing rule matches against, not something a human reads. annotations are the reverse — human-readable context, with templating access to the alert's own labels and the value that triggered it, meant to be read by whoever gets paged.

The for: Duration and Alert States

for: 10m means the condition has to stay true continuously for ten minutes before the alert actually fires — a deliberate guard against a single brief blip triggering a page. An alert moves through real, named states: pending (the condition just became true, but for hasn't elapsed yet), firing (the full duration has passed, and it's now been sent to Alertmanager), and resolved (the condition is no longer true — Alertmanager can notify on this too, so a resolved incident doesn't require someone to manually confirm it's over).

Alertmanager — Routing, Grouping, and Silencing

• Routing — a tree of label matchers deciding which receiver (Slack, email, PagerDuty, ...) a given alert goes to, based on its own labels.
• Grouping — multiple alerts sharing similar labels get bundled into one notification, rather than flooding a channel with dozens of separate messages during one widespread incident — directly answering cloud1-9's own alert-fatigue warning.
• Silencing — temporarily mutes alerts matching a given label set, for planned maintenance or a known ongoing issue, without touching the underlying rule itself.

A Routing Tree Example

group_wait is how long Alertmanager waits after the first alert in a new group before sending the initial notification, giving related alerts a chance to arrive and be bundled together. group_interval controls how long to wait before sending an update about new alerts joining an already-notified group. repeat_interval is how long to wait before re-sending a notification for a group that's still firing, unresolved — these three timers together are what actually implement grouping's own noise reduction, not just a conceptual idea.

Avoiding Alert Fatigue — Directly Building On cloud1-9

cloud1-9 named the problem without the tooling; this chapter's routing, grouping, and the for duration together are the concrete fix. Route by real urgency — not everything to the same pager. Group related alerts so one incident produces one notification, not fifty. And before ever setting severity: critical, ask the honest question: if this fires at 3am, does someone genuinely need to get out of bed for it? An alert that doesn't clear that bar belongs at a lower severity, or shouldn't page a human at all.

Hands-On Exercises

Metrics are done. This chapter moves to the second pillar obs1-1 named — logs — and delivers the real tooling cloud2-4's own correlation-ID material assumed was already in place.

Structured vs. Unstructured Logs

Unstructured logs are free text, meant for a human reading them one line at a time — extracting user_id or duration_ms afterward means writing a regex against the message, fragile and quick to break the moment someone tweaks the wording. Structured logs emit their fields directly, as data — every field is queryable and filterable from the moment it's written, with no parsing guesswork required downstream.

Log Aggregation — Why You Need a Central System

In a system with dozens of services across dozens of instances, logs scattered across that many individual machines are practically unsearchable during a real incident — nobody is going to SSH into thirty containers one at a time while something is on fire. A log aggregation system centralizes logs from every source into one searchable place, which is exactly the capability cloud2-4's own "first five minutes" incident-response technique assumed was already sitting there, ready to use.

Loki vs. the ELK/EFK Stack — Two Different Philosophies

ELK (Elasticsearch, Logstash, Kibana) — or EFK, swapping in Fluentd/Fluent Bit — indexes the full text of every log line in Elasticsearch, enabling powerful free-text search across any word, anywhere, at real storage and compute cost. Loki takes a deliberately different approach: it doesn't index log content at all — only labels, the exact same concept as obs1-2's own Prometheus labels, applied here to logs instead of metrics. The actual log lines are stored compressed, cheaply, and only scanned — not separately indexed — within whichever label-selected stream a query has already narrowed down. This is a deliberate design choice, not a coincidence: Loki is built by the same team behind Grafana specifically to pair with Prometheus's own label-based philosophy, at a fraction of the resource cost full-text indexing requires.

LogQL — Loki's Own Query Language

{job="checkout-service"} is the label selector — cheap, indexed, exactly like a PromQL selector from obs1-4. |= "timeout" is a line filter, applied only within that already-narrowed stream — not indexed, but only scanning a small, pre-selected subset rather than every log line ever written. | json parses each structured line and exposes its fields for further filtering, as in the duration_ms > 5000 example — turning a structured field written once at log time into something directly queryable later.

Correlation IDs — Delivering On cloud2-4's Own Material

cloud2-4 named correlation IDs as the technique for tracing one request across multiple services' own logs, conceptually. Here's the concrete mechanism: a unique ID is generated once, at the edge — typically an API gateway — and passed along on every downstream service call, usually as an HTTP header, then included as a structured field in every log line each service emits while handling that request.

That single query, matching every job, filters down to every log line — across every service that touched this one request — sharing that exact request_id. This is the concrete technique that turns "find every log line related to this one failing request, across the whole system" from a conceptual goal into something you can actually run. obs1-8's traces are the even more powerful version of this same underlying idea — correlating not just log lines, but full per-service timing across the whole request.

Hands-On Exercises

obs1-7 closed with a promise: traces are the more powerful version of the same correlation idea logs only partially deliver. This chapter is where that becomes concrete — the third pillar, and genuinely new material nothing else on this site has covered before now.

What a Trace Actually Is

A trace represents one request's full journey across every service it touches. It's built from spans — each span represents one unit of work: one service handling the request, or one specific operation inside it, like a database query. A span has a name, a start time, a duration, and a set of key-value attributes, similar in spirit to a metric's own labels.

Spans — The Building Block

Every span has a unique span ID, and — except the very first, root span — a parent span ID, forming a tree that reconstructs the exact call graph a request actually took. Every span belonging to the same request shares one trace ID, the thread tying the whole tree together.

This is obs1-1's own checkout example, made literal. "3.8 of the 4 seconds spent waiting on inventory" isn't a conclusion someone had to piece together from separate logs anymore — it's a visible span in this exact tree, with its own duration sitting right there.

Trace Context Propagation

For spans created in separate services to link into one trace, the trace ID — and the current span ID, to establish the parent-child relationship — has to travel along with every downstream call, usually as an HTTP header. This is mechanically the same idea as obs1-7's own correlation ID, formalized into a real, standardized format: the W3C Trace Context traceparent header, rather than every team inventing its own ad-hoc header.

Four dash-separated parts: version, trace ID, the parent span's own ID, and trace flags. Every hop that receives this header, creates its own span as a child of that parent ID, and forwards an updated version downstream — the exact mechanism that turns separate services' separate spans into one connected tree.

OpenTelemetry — A First Look at Instrumenting Your Code

OpenTelemetry (OTel) is the vendor-neutral standard for generating this telemetry — an SDK application code calls directly to create spans, which are then exported to a tracing backend. obs1-9 covers it properly; here's the shape of it:

Many frameworks and libraries support auto-instrumentation — OTel can automatically create spans for common operations (incoming HTTP requests, outgoing calls, database queries) with no manual code changes at all, while manual spans like the one above cover anything genuinely business-specific a generic instrumentation library couldn't know to track on its own.

Jaeger — Storing and Visualizing Traces

Jaeger receives the spans instrumented services export, stores them, and provides a UI to search traces — by service, operation, duration, tags — and to visualize any single trace as a waterfall/timeline view, exactly like the tree above but interactive. It's the tracing pillar's own equivalent of what Grafana is for metrics: a query-and-visualization layer sitting on top of the actual data. And as obs1-5 already noted, Grafana itself can visualize Jaeger (or Tempo, its close relative) traces directly as a data source, right alongside metrics and logs in the same dashboard.

Hands-On Exercises

obs1-1's own tip-box promised this chapter would be about correlation, not just collection. obs1-8 previewed the SDK's own shape. This is where both promises land — OpenTelemetry as the vendor-neutral standard unifying all three pillars under one instrumentation approach.

The Problem OpenTelemetry Solves — Vendor Lock-In and Fragmented Instrumentation

Before OTel, each observability vendor — Datadog, New Relic, and others — shipped its own proprietary instrumentation SDK. Switching vendors meant re-instrumenting an entire codebase from scratch. OpenTelemetry, a CNCF project, is the industry's answer: one vendor-neutral standard covering metrics, logs, and traces together, with a shared wire protocol — OTLP — for exporting telemetry to any compatible backend. Instrument once, send anywhere; swapping Jaeger for a different tracing backend, or Prometheus for a different metrics store, becomes an exporter configuration change, not an application code change.

The Three Signals, One SDK

OTel's own vocabulary calls metrics, logs, and traces signals — mirroring obs1-1's own "three pillars" framing, just OTel's specific term for it. The genuine practical payoff: one SDK exposes a tracer, a meter, and a logger side by side, sharing one consistent API — rather than three separate libraries, each with its own conventions, that happen to be used together.

The OpenTelemetry Collector — The Piece That Ties Everything Together

The Collector is a separate, standalone process that receives telemetry over OTLP from instrumented applications, can process or filter it, and exports it to one or more backends. This is the concrete mechanism behind obs1-1's own correlation promise: one pipeline, configured once, feeding Ch.2-4's Prometheus, Ch.7's Loki, and Ch.8's Jaeger from the exact same instrumented application code, with no separate integration work per backend.

Auto-Instrumentation Revisited

obs1-8 previewed this: many OTel language SDKs can auto-instrument common frameworks — HTTP servers and clients, database drivers — with zero code changes, automatically producing spans (and, for the fuller picture this chapter adds, metrics and logs too) for the "boring 80%" of a system's own telemetry. Manual instrumentation, like obs1-8's own process_payment span example, is then reserved specifically for business-specific logic auto-instrumentation could never know to track on its own.

Correlating Signals — The Real Payoff

Because metrics, logs, and traces all flow through the same SDK and the same Collector, they can share correlation context automatically. A log line emitted while a span is active is automatically enriched with that span's own trace ID — the exact same idea obs1-7's own request_id field manually engineered, now produced for free by the underlying mechanism itself:

Metrics get their own version of this same idea via exemplars — a real Prometheus/Grafana feature that attaches an example trace ID directly to a specific data point on a histogram, letting you jump straight from "this latency spike, right here on the graph" to "here's an actual trace from exactly that moment." This is what obs1-1's own tip-box meant by correlation being the real goal, not just having all three pillars collected somewhere.

Hands-On Exercises

k8s2-7's own Observability chapter covered Metrics Server, Prometheus scraping, and Grafana at a deliberately light touch, closing with an honest note that a fuller course existed for anyone who needed to go deeper. This is that chapter — applying everything from Ch.2 through Ch.9 specifically to Kubernetes' own genuinely different operating conditions.

Why Kubernetes Needs Its Own Observability Chapter

obs1-3 already named the core problem: a hand-maintained, static target list breaks the moment instances scale up, scale down, or get rescheduled — exactly what happens continuously in Kubernetes, where pods are genuinely ephemeral and routinely get new IPs. And this is distinct from k8s1-11's own liveness/readiness probes — probes are Kubernetes' own built-in mechanism for restarting or routing around an unhealthy pod; this chapter is about actually observing metrics, logs, and traces from those same pods, a separate and complementary concern.

kube-state-metrics — Metrics About Kubernetes Objects Themselves

obs1-3's node_exporter reports host-level metrics — CPU, memory, disk of the underlying machine. kube-state-metrics reports something genuinely different: the state of Kubernetes API objects themselves — how many pods a Deployment currently has running versus desired, how many replicas are unavailable, pod restart counts, whether a PersistentVolumeClaim is bound. This is directly the same "desired vs. actual state" theme k8s1-2's own reconciliation loop and k8s1-5's ReplicaSet material already established, now exposed as real, queryable metrics.

A genuinely useful alert source on its own — a Deployment with unavailable replicas is a direct, concrete instance of obs1-6's own alert-rule material, applied here to Kubernetes' own object state rather than an application's business metrics.

The Prometheus Operator — Managing Prometheus the Kubernetes Way

Running Prometheus by hand inside Kubernetes — a raw Deployment plus a ConfigMap holding prometheus.yml — works, but every new scrape target means manually editing that config and reloading it. The Prometheus Operator introduces Kubernetes-native Custom Resources that let Prometheus's own configuration be managed declaratively, the same way any other Kubernetes object is defined per k8s1-4's own YAML-manifest material — rather than as one hand-edited file.

ServiceMonitors — Declarative Scrape Configuration

A ServiceMonitor declares "scrape any Service matching these labels, on this port, at this interval." The Prometheus Operator watches for ServiceMonitor objects and automatically reconfigures the underlying Prometheus to match, live — no manual reload required. This is Kubernetes' own real fix for obs1-3's static-target problem: instead of a generic kubernetes_sd_config still requiring manual relabeling rules, each application team declares its own scrape intent right alongside its own Service definition, reviewed and version-controlled the exact same way obs1-5 already argued dashboards should be, and echoing k8s2-9's own GitOps pattern.

Logs and Traces in Kubernetes — A Brief Practical Note

Containers write logs to stdout/stderr by convention; a node-level agent — Promtail for Loki, or an OTel Collector deployed as a DaemonSet (k8s2-2's own one-pod-per-node pattern) — tails those logs and ships them centrally, automatically attaching pod, namespace, and container labels along the way. Traces work similarly: an OTel Collector, again commonly run as a DaemonSet or a sidecar, receives OTLP from instrumented pods and forwards it to Jaeger, with the same Kubernetes metadata attached automatically as span attributes. This is deliberately a light touch — the bulk of this chapter's genuinely new material is the metrics side (kube-state-metrics, the Operator, ServiceMonitors); logs and traces in Kubernetes are mostly Ch.7/8's own tools, deployed in a Kubernetes-shaped way.

Hands-On Exercises

The final chapter. This capstone wires Prometheus, Grafana, Loki, Jaeger, Alertmanager, and the OpenTelemetry Collector together into one real stack, running the exact checkout-service scenario obs1-1 opened this course with — and closes by walking one real incident through it end to end, using all three pillars the way an actual investigation would.

The Architecture — One Pipeline, Three Backends

Four OTel-instrumented services — gateway, auth-service, checkout-service, inventory-service — running in Kubernetes. Each exports telemetry via OTLP to an OpenTelemetry Collector running as a DaemonSet (obs1-10). The Collector fans out to Prometheus (metrics), Loki (logs), and Jaeger (traces). Grafana sits on top, querying all three as data sources (obs1-5). Alertmanager receives fired alerts from Prometheus and routes them (obs1-6). One pipeline, configured once, feeding every backend this course built.

Step 1 — Discovery via ServiceMonitors

One ServiceMonitor, matching a shared part-of: checkout-stack label across all four services — obs1-10's own real fix for hand-maintaining a static target list as these pods scale and get rescheduled.

Step 2 — Instrumentation via OpenTelemetry

Auto-instrumentation covers incoming/outgoing HTTP calls and database queries across all four services with no code changes. checkout-service adds one manual span for its own business logic — the exact obs1-8/obs1-9 pattern:

Step 3 — Metrics & Dashboards

Every service exposes http_requests_total (counter) and http_request_duration_seconds (histogram) via auto-instrumentation, plus kube_deployment_status_replicas_unavailable from kube-state-metrics (obs1-10). One Grafana dashboard, built with a $service template variable (obs1-5), covers all four services with three panels: request rate, error rate (obs1-4's own percentage-of-5xx pattern), and p99 latency via histogram_quantile() — one dashboard, not four.

Step 4 — Alerting

obs1-6's own rule, reused directly, now grouped by (job) so each service's error rate is tracked independently. Routed critical → PagerDuty, warning → Slack, grouped by alertname/job so a widespread incident produces one notification, not one per affected instance.

Step 5 — Logs

Structured JSON (obs1-7), with trace_id attached automatically by the OTel SDK (obs1-9) rather than a hand-engineered correlation field. Any log line from any of the four services can be found via one LogQL query filtered on that exact trace_id.

Step 6 — Traces

obs1-1's own opening example, now a real trace flowing through the actual pipeline built across this entire course — 3.8 of 4.1 seconds spent in inventory-service, visible directly in Jaeger's own waterfall view.

A Real Incident, Walked End to End

1. Alertmanager fires HighErrorRate for checkout-service, routed to Slack (severity: warning at first, since error rate hasn't yet crossed into timeouts becoming outright 5xx failures).
2. Grafana's dashboard, filtered to $service=checkout-service, shows the p99 latency spike and pinpoints the time window.
3. Explore mode against Loki, filtered to that time window and service, surfaces the "inventory reservation timeout" log line and its trace_id.
4. Jaeger, queried by that exact trace ID, shows the full tree above — inventory-service's own db-query span is where the time actually went, not checkout-service itself.

Metrics said something was wrong; logs said what; traces said exactly where — obs1-1's own three-question framing, now answered by a real, working pipeline rather than a conceptual walkthrough.

Hands-On Exercises