Exercise 3: The Payments Service That Drops the request_id — Possible Solution ==================================================================== -- What breaks during an incident investigation -- -- -- A query like {job=~".+"} | json | request_id = "abc-123" -- the -- chapter's own cross-service correlation pattern -- depends entirely -- on every service in the request's path actually including that -- exact field in its own log lines. Because the payments service was -- recently rewritten and never added request_id to its logging, that -- query would correctly return every relevant log line from the API -- gateway and the auth service, but NONE from payments, even though -- payments genuinely handled this same request and may well be where -- the actual problem occurred. From the investigator's point of view, -- the trail simply goes cold right after auth -- there's no error, no -- warning, nothing indicating that a whole service's worth of -- relevant logs is silently missing from the result. It looks exactly -- like payments either wasn't involved or didn't log anything -- noteworthy, when the real explanation is that its logs exist but -- are simply unreachable by request_id. -- Why this is easy to miss until it matters -- -- -- The gap is invisible under normal conditions -- payments is still -- logging normally, its logs are still searchable by other fields -- (its own job label, timestamp, log level), and nobody investigating -- routine, non-urgent logs would necessarily notice the missing field -- unless they were specifically trying to correlate by request_id. -- The problem only becomes visible during an actual incident, exactly -- when someone is under time pressure trying to trace one specific -- failing request end to end -- the worst possible moment to discover -- that one service in the chain was quietly excluded from -- correlation this whole time. This matches the chapter's own -- warn-box precisely: one service dropping the ID breaks the whole -- trail silently, with the gap typically only surfacing when it's -- least convenient. WHY THIS WORKS AS AN ANSWER ------------------------------ This traces exactly what the correlation query would and wouldn't return given the described gap, explaining why the missing service's logs vanish from the trail without any visible error, then explains why this specific kind of gap tends to go unnoticed until an incident actually depends on it.