Exercise 3: A Sampling Strategy That Still Guarantees Errors and Slow Requests — Possible Solution ==================================================================== Proposed strategy: tail-based sampling with error/latency overrides - Trace every request as normal, but only make the final keep-or-discard decision AFTER the request completes (this is what distinguishes "tail-based" sampling from deciding upfront, before the outcome is known). - Always KEEP (export and store) the full trace if the request resulted in an error, or if its total duration exceeded a defined threshold (e.g. the service's own p95 latency). - For all other, ordinary successful/fast requests, keep only a small fixed percentage (e.g. 5-10%) as a representative sample, discarding the rest. -- Why this is a reasonable tradeoff, not a loss of visibility -- -- -- The overhead the team is seeing at peak traffic comes primarily -- from exporting and storing every single trace, most of which are -- for perfectly ordinary, successful, fast requests -- the ones -- almost never actually looked at during an investigation. Under -- this strategy, exactly the requests someone would actually want a -- full trace for during an incident -- the errors, and the -- suspiciously slow ones -- are still guaranteed to be fully traced -- and available, 100% of the time, with none of that visibility -- lost. What's actually reduced is only the volume of traces for -- requests that were already fine, which were rarely if ever the -- ones anyone needed to look at afterward anyway. The small -- percentage of ordinary-request traces still kept is enough to spot -- broader trends or confirm what "normal" latency looks like for -- comparison, without needing to store and export a full trace for -- every single uneventful request at peak volume. WHY THIS WORKS AS AN ANSWER ------------------------------ This proposes a concrete, named strategy (tail-based sampling with error/latency overrides) rather than a vague "sample some requests" answer, and explains specifically why discarding most successful requests doesn't cost real investigative value, since those are rarely the requests anyone actually needs a trace for.