Grafana — Dashboards & Visualization

obs1-4 gave you PromQL. This chapter gives you somewhere to put it — Grafana, the visualization layer this course pairs with Prometheus. And in keeping with obs1-1's own opening warn-box, this chapter is honest about what a dashboard, no matter how well-built, actually is and isn't.

What Grafana Actually Is — A Visualization Layer, Not a Data Store

Grafana itself stores no metrics. It's a query-and-render frontend that connects to one or more data sources — Prometheus here, but also Loki for logs (obs1-7) and Jaeger/Tempo for traces (obs1-8) — and turns their query results into panels. This separation matters in practice: losing Grafana loses no actual data, since everything it displays lives in the underlying data source, not in Grafana itself. It also means one Grafana dashboard can show metrics, logs, and traces side by side, sourced from entirely different systems.

Data Sources — Connecting Grafana to Prometheus

A data source is configured once — a name, a type (Prometheus), and the URL Grafana should query. Once that connection exists, every panel in every dashboard can query it directly using ordinary PromQL, exactly as written in obs1-4.

Panels — The Basic Building Block

A panel is one visualization, backed by one or more queries. Common types: a time series graph, a single stat/number, a gauge, a table, and a heatmap — genuinely useful specifically for visualizing a histogram's own bucket distribution over time.

That's the exact error-rate query from obs1-4, dropped straight into a panel — a panel is nothing more than a chosen visualization wrapped around a PromQL query you already know how to write.

Template Variables — One Dashboard, Many Contexts

$service is a template variable — a dropdown at the top of the dashboard, typically populated by a label_values() query against Prometheus itself, that gets substituted into every panel's query at render time. One dashboard, built once, can then serve every service sharing that shape of metric — rather than hand-building a near-identical dashboard per service.

Dashboards as Code — Provisioning

Building a dashboard by clicking through Grafana's own UI works, but doesn't scale and isn't reviewable the way real code is. Grafana dashboards can instead be defined as JSON, checked into version control alongside application code — the same review-and-history discipline git1-3 already established for everything else. Provisioning is what makes this practical: Grafana can automatically load dashboard JSON and data source configuration from files at startup, rather than requiring anyone to manually recreate them through the UI — a light echo of k8s2-9's own GitOps pattern, applied here to dashboards instead of Kubernetes manifests.

An Honest Note — Dashboards Are Still Monitoring, Not Observability

obs1-1's own warn-box applies here directly: a dashboard, however well-designed, only ever answers the questions its panels were built to answer in advance. Grafana's Explore mode is its own genuine answer to that limit — an interface for running ad-hoc PromQL queries against a data source without needing a saved panel first, letting you follow a brand-new question on the spot during an incident rather than being limited to whatever panels already exist.

Hands-On Exercises