Prometheus Architecture & Scraping

Observability

Chapter 3 · Prometheus Architecture & Scraping

obs1-2 established what a metric looks like once it exists. This chapter covers how it actually gets from an application into Prometheus in the first place — the architectural choice that shapes nearly everything else about how the system is operated.

The Pull Model — Prometheus Comes to You

Prometheus's core design decision: it scrapes (pulls) metrics by periodically sending an HTTP GET to a /metrics endpoint each target exposes, in the exposition format from obs1-2. Applications don't send their metrics anywhere — they just expose them on an endpoint and wait to be asked, on whatever schedule Prometheus's own configuration decides.

Why Pull? — Real Advantages

Centralized control — scrape frequency and target lists live in one place (Prometheus's own config), not scattered across every application's own settings.
A failed scrape is itself a signal — if Prometheus can't reach a target, that's meaningful information ("this target is unreachable") distinct from "this target reported zero" — something a push-based system can't easily distinguish from silence.
Simpler service discovery — Prometheus decides who to scrape and finds them; individual services never need to know Prometheus's own address.
Trivially testable locally — curling a /metrics endpoint shows exactly what Prometheus would see, with no special tooling required.

When Pull Doesn't Fit — Pushgateway

A short-lived batch job can finish and exit before any scheduled scrape would ever reach it — the pull model has nothing to pull from once the process is gone. The Pushgateway is the sanctioned exception: a batch job pushes its final metrics to the Pushgateway once, right before exiting, and Prometheus then scrapes the Pushgateway itself, like any other ordinary target.

Pushgateway is an exception, not a general push mechanism

Using the Pushgateway for regular, long-running services defeats the entire point of the pull model's own advantages — a failed scrape stops meaning "the target is down" once every value is being relayed through an intermediary. It exists specifically for short-lived jobs that genuinely can't be scraped directly, nothing broader.

Exporters — Getting Metrics Out of Things That Don't Speak Prometheus

Most systems — MySQL, Redis, the Linux kernel itself, older applications — don't natively expose a Prometheus-format /metrics endpoint. An exporter is a small adapter process that queries the underlying system in whatever native way it supports, then re-exposes that data as an ordinary Prometheus-format endpoint for scraping. This is exactly the missing piece cloud1-8's own terminology-level coverage never got into.

node_exporter — host-level metrics: CPU, memory, disk, network, straight from the OS
mysqld_exporter / redis_exporter — connect to the database using its own native protocol, re-expose the results as Prometheus metrics
blackbox_exporter — actively probes external endpoints (HTTP, TCP, ICMP) and reports reachability/latency as metrics

Configuring Scrape Targets — prometheus.yml

scrape_configs:
  - job_name: 'node'
    scrape_interval: 15s
    static_configs:
      - targets: ['localhost:9100']

  - job_name: 'my-app'
    static_configs:
      - targets: ['app1:8080', 'app2:8080']
  

Each job_name groups a set of targets sharing the same scrape configuration. Every scraped sample is automatically tagged with a job label (and an instance label identifying which specific target), giving every metric a built-in "which service, which instance" dimension for free.

Service Discovery — Beyond Hardcoded Targets

A static, hand-maintained target list breaks the moment instances scale up, scale down, or get replaced — exactly what happens continuously in Kubernetes or a cloud autoscaling group. Prometheus supports service discovery mechanisms — kubernetes_sd_config, ec2_sd_config, consul_sd_config, among others — that automatically discover the current set of scrape targets directly from the underlying platform, rather than requiring anyone to keep a list in sync by hand. obs1-10 covers Kubernetes' own version of this in depth, via ServiceMonitors and the Prometheus Operator.

Retention — Local Storage Isn't Forever

Prometheus stores scraped samples on local disk in its own time-series database, with a configurable retention window — 15 days is a common default. This is a deliberate scoping decision, not an oversight: Prometheus is built for reasonably recent operational data, not as a long-term archive. For genuinely long-term retention or a unified view across multiple Prometheus instances, remote-writing to a dedicated long-term storage backend (Thanos, Cortex, Mimir) is the standard real-world answer — worth naming honestly here as beyond this course's own scope, a deliberate line drawn rather than an oversight.

	Pull (Prometheus's default)	Push
Who initiates	The monitoring system, on its own schedule	The application, whenever it decides to
A silent target	Detected directly — the scrape itself fails	Ambiguous — is it down, or just not pushing right now?
Config location	Centralized, in the monitoring system	Scattered across every application

The up metric is one of the most useful things Prometheus gives you for free

Every scrape target automatically gets an up metric — 1 if the last scrape succeeded, 0 if it didn't. This single built-in metric, direct proof of the pull model's own "a failed scrape is itself a signal" advantage, is often the very first thing worth checking during an incident.

Hands-On Exercises

Exercise 1

Explain, using this chapter's own reasoning, why a failed scrape is more informative than a push-based system simply not receiving any data for a while.

📄 View solution

Exercise 2

A nightly batch job runs for 45 seconds and then exits. Explain why scraping it directly wouldn't reliably work, and what the correct Prometheus-native solution is.

📄 View solution

Exercise 3

Write a scrape_configs entry for a job named "redis" scraping targets at redis1:9121 and redis2:9121 every 30 seconds, and explain why redis_exporter is needed here rather than scraping Redis directly.

📄 View solution

Chapter 3 Quick Reference

Prometheus pulls (scrapes) metrics from a /metrics HTTP endpoint on a schedule, rather than applications pushing to it
A failed scrape is itself meaningful — the built-in up metric reports 1 (success) or 0 (failure) per target
Pushgateway — the sanctioned exception, for short-lived batch jobs only, not general-purpose push
An exporter adapts a system that doesn't natively speak Prometheus into a scrapeable /metrics endpoint (node_exporter, mysqld_exporter, blackbox_exporter)
scrape_configs in prometheus.yml defines jobs, targets, and scrape intervals; job/instance labels are added automatically
Service discovery (kubernetes_sd_config, etc.) replaces hand-maintained static target lists in dynamic environments
Local retention is intentionally limited (often ~15 days); Thanos/Cortex/Mimir are the standard long-term-storage answer, out of this course's own scope