Observability in Kubernetes

k8s2-7's own Observability chapter covered Metrics Server, Prometheus scraping, and Grafana at a deliberately light touch, closing with an honest note that a fuller course existed for anyone who needed to go deeper. This is that chapter — applying everything from Ch.2 through Ch.9 specifically to Kubernetes' own genuinely different operating conditions.

Why Kubernetes Needs Its Own Observability Chapter

obs1-3 already named the core problem: a hand-maintained, static target list breaks the moment instances scale up, scale down, or get rescheduled — exactly what happens continuously in Kubernetes, where pods are genuinely ephemeral and routinely get new IPs. And this is distinct from k8s1-11's own liveness/readiness probes — probes are Kubernetes' own built-in mechanism for restarting or routing around an unhealthy pod; this chapter is about actually observing metrics, logs, and traces from those same pods, a separate and complementary concern.

kube-state-metrics — Metrics About Kubernetes Objects Themselves

obs1-3's node_exporter reports host-level metrics — CPU, memory, disk of the underlying machine. kube-state-metrics reports something genuinely different: the state of Kubernetes API objects themselves — how many pods a Deployment currently has running versus desired, how many replicas are unavailable, pod restart counts, whether a PersistentVolumeClaim is bound. This is directly the same "desired vs. actual state" theme k8s1-2's own reconciliation loop and k8s1-5's ReplicaSet material already established, now exposed as real, queryable metrics.

A genuinely useful alert source on its own — a Deployment with unavailable replicas is a direct, concrete instance of obs1-6's own alert-rule material, applied here to Kubernetes' own object state rather than an application's business metrics.

The Prometheus Operator — Managing Prometheus the Kubernetes Way

Running Prometheus by hand inside Kubernetes — a raw Deployment plus a ConfigMap holding prometheus.yml — works, but every new scrape target means manually editing that config and reloading it. The Prometheus Operator introduces Kubernetes-native Custom Resources that let Prometheus's own configuration be managed declaratively, the same way any other Kubernetes object is defined per k8s1-4's own YAML-manifest material — rather than as one hand-edited file.

ServiceMonitors — Declarative Scrape Configuration

A ServiceMonitor declares "scrape any Service matching these labels, on this port, at this interval." The Prometheus Operator watches for ServiceMonitor objects and automatically reconfigures the underlying Prometheus to match, live — no manual reload required. This is Kubernetes' own real fix for obs1-3's static-target problem: instead of a generic kubernetes_sd_config still requiring manual relabeling rules, each application team declares its own scrape intent right alongside its own Service definition, reviewed and version-controlled the exact same way obs1-5 already argued dashboards should be, and echoing k8s2-9's own GitOps pattern.

Logs and Traces in Kubernetes — A Brief Practical Note

Containers write logs to stdout/stderr by convention; a node-level agent — Promtail for Loki, or an OTel Collector deployed as a DaemonSet (k8s2-2's own one-pod-per-node pattern) — tails those logs and ships them centrally, automatically attaching pod, namespace, and container labels along the way. Traces work similarly: an OTel Collector, again commonly run as a DaemonSet or a sidecar, receives OTLP from instrumented pods and forwards it to Jaeger, with the same Kubernetes metadata attached automatically as span attributes. This is deliberately a light touch — the bulk of this chapter's genuinely new material is the metrics side (kube-state-metrics, the Operator, ServiceMonitors); logs and traces in Kubernetes are mostly Ch.7/8's own tools, deployed in a Kubernetes-shaped way.

Hands-On Exercises