Exercise 3: Two Separate Problems With a user_id-Labeled Latency Histogram — Possible Solution ==================================================================== -- Problem 1: Cardinality explosion -- -- -- user_id is exactly the kind of unbounded value the chapter's own -- warn-box names directly -- a real production system might have -- thousands or millions of distinct users, and every distinct user_id -- value creates a genuinely SEPARATE time series for every single -- histogram bucket that metric has. A histogram with, say, 10 buckets -- and 100,000 distinct users would explode into roughly a million -- individual time series from one instrumented metric alone, -- multiplying Prometheus's storage and memory usage by that same -- factor. This is a real, documented cause of production Prometheus -- outages, not a theoretical concern. -- Problem 2: This particular plan wouldn't actually need aggregation -- across instances to already be broken, but assuming user_id were -- somehow made safe (e.g. bucketed into a small number of user -- tiers instead), the aggregation plan itself is still fine BECAUSE -- it's a histogram, not a summary -- -- -- The colleague's plan was actually to use a histogram specifically -- (not a summary), which is the right instinct for cross-instance -- aggregation: histogram bucket counts genuinely can be summed -- across every instance, and histogram_quantile() can then compute a -- real fleet-wide p99 from the combined buckets, exactly as the -- chapter describes. If the colleague had instead reached for a -- summary metric (client-side quantiles computed per instance), -- THAT would have been the second real problem -- per-instance -- summary quantiles can't be meaningfully averaged together into a -- genuine fleet-wide p99, since each instance's own summary quantile -- was already computed from that one instance's own subset of -- requests, with no way to recombine them after the fact. -- Combined verdict -- -- -- The cardinality problem (an unbounded user_id label) is real and -- would need fixing regardless of metric type. The -- histogram-vs-summary concern only becomes a second real problem if -- the metric were built as a summary instead of a histogram -- so the -- full answer names both risks explicitly, rather than treating them -- as one single issue. WHY THIS WORKS AS AN ANSWER ------------------------------ This separates the two genuinely distinct risks the exercise asks for -- an unbounded label causing cardinality explosion, and the histogram-vs-summary aggregation distinction from the chapter's own compare-table -- rather than conflating them into one vague warning, and is explicit about which risk applies regardless of metric type and which one is conditional on the metric type chosen.