Exercise 1: node_exporter vs. kube-state-metrics for a Crashing Pod — Possible Solution ==================================================================== Scenario: a pod is repeatedly crashing and restarting. -- What node_exporter would report -- -- -- node_exporter reports on the underlying MACHINE, not on any -- specific pod's own behavior. If the crashing pod itself isn't -- consuming unusual CPU/memory/disk on the node it happens to be -- scheduled on, node_exporter's own metrics might show nothing -- unusual at all -- the node itself could look perfectly healthy -- even while one specific pod on it is in a crash loop. At most, -- node_exporter might indirectly reflect elevated resource usage IF -- the crash loop itself is resource-intensive (e.g. the pod -- repeatedly tries to allocate a lot of memory right before it -- crashes), but that's incidental, not something node_exporter is -- actually designed to surface. -- What kube-state-metrics would report -- -- -- kube-state-metrics reports directly on the Kubernetes API's own -- knowledge of this pod's state -- specifically, metrics like pod -- restart counts (kube_pod_container_status_restarts_total) climbing -- rapidly, and the pod's own status.phase or container status -- reflecting CrashLoopBackOff. This is exactly the kind of state -- -- "does this specific Kubernetes object's actual condition match -- what was declared/expected" -- that kube-state-metrics is built to -- expose, and it's the metric an alert rule would actually be built -- against to catch this incident (a rising restart count, or time -- spent in a non-Running state). -- The core distinction -- -- -- node_exporter answers "is the MACHINE healthy" -- largely blind to -- one specific pod's own crash-looping behavior unless it happens to -- also stress the node's resources. kube-state-metrics answers "is -- THIS SPECIFIC KUBERNETES OBJECT in the state it's supposed to be -- in" -- directly and immediately reflecting the crash loop as -- exactly the kind of desired-vs-actual-state mismatch the chapter -- names as kube-state-metrics's own core purpose. WHY THIS WORKS AS AN ANSWER ------------------------------ This works through the same concrete incident from both tools' own actual scope (machine-level vs. Kubernetes-object-level), showing specifically why node_exporter would likely miss this incident while kube-state-metrics would directly and reliably surface it.