Exercise 3: Why the 30%/3-of-10 Match Mattered — Possible Solution ==================================================================== WHY THE MATCH WAS SIGNIFICANT ------------------------------ Per this chapter, checking each instance's version "confirms exactly the mid-rollout pattern above - 3 of 10 instances on the new version. 3 out of 10 is 30%, matching the failure rate almost exactly." A coincidental match between the proportion of requests failing and the proportion of instances running new code is strong, near-conclusive evidence that the failures are specifically tied to which version handles a given request - if the cause were unrelated to version skew, there would be no reason for the failure rate to line up numerically with the new-version instance count at all. WHY THIS WAS BETTER THAN JUST NOTICING "IT STARTED AFTER A DEPLOY" ------------------------------ Timing alone (failures starting right after a deploy) is suggestive but not conclusive - plenty of things could coincidentally start around the same time as a deploy. The specific numeric match between failure rate and new-version instance proportion is a much stronger, quantitative confirmation that ties the symptom directly to the rollout's own instance mix, not just its timing. THE ACTUAL ROOT CAUSE ------------------------------ Per this chapter, "the new version's code expects a display_name field that a migration was meant to backfill onto every existing user record - but the backfill is still running and only partially complete. Requests landing on new-version instances fail specifically for the subset of accounts the backfill hasn't reached yet." The root cause was new code deployed ahead of a still-incomplete backfill migration - exactly the lockstep-timing mistake the expand-contract pattern exists to prevent. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains why the specific numeric correspondence (not just the timing) was strong evidence, and correctly identifies the underlying cause as an incomplete backfill combined with code deployed ahead of it, connecting back to this chapter's own expand-contract discussion.