Deployment-Related Symptoms: Version Skew & Migration Failures

Chapter 2 named "100% of requests fail, suddenly" as the pattern most likely to point at a deployment. This chapter gives that heuristic its fullest treatment — what actually happens during the window a rolling deployment is in progress, why a database migration can fail independently of the code deploy that depends on it, and a concrete technique for confirming a deployment is the cause rather than just suspecting it.

Version Skew: Old and New Code, Running at the Same Time

A rolling or blue-green deployment inevitably passes through a window where old and new code run simultaneously, across different server instances. If the new version changes an API contract in a way that isn't backward- or forward-compatible — a renamed field, a newly required field, a changed response shape — requests can fail unpredictably depending on which specific instance happens to handle them during that window. This applies to service-to-service communication too, not just client requests: a frontend expecting the new shape can just as easily hit an old backend instance, or the reverse.

This directly explains a genuinely confusing symptom: "it works sometimes and fails other times, right after a deploy, with no code changes since" — because which instance answers a given request is effectively random during the rollout, and each instance is running one version or the other, never a blend.

Recognizing Version Skew From the Outside

Directly checking each instance's own reported version settles this immediately:

Three of ten instances already on the new version, seven still on the old — a rollout genuinely mid-flight. In a container-orchestrated environment, checking each running pod's own image tag serves the same purpose.

Database Migration Failures

A schema migration can fail partway through, or succeed on the database while the application code deployed alongside it doesn't actually match what the migration has finished doing yet. A genuinely common real mistake: deploying code that expects a new field before the migration that backfills it has actually completed everywhere — or the reverse, old code still running against a schema that's already been changed underneath it.

A Genuine Technique: Bisecting by Time

When an error rate shows a sudden, sharp step-change at one specific timestamp — not a gradual climb — checking exactly what deployed or migrated at that precise moment is almost always faster than examining application code in the abstract. The deployment history and migration log are themselves evidence, in exactly the same spirit as this subject's own "check first, don't guess" theme.

Working Example: The 30% "Field Not Found" Ticket

A fresh ticket: right after this afternoon's deploy, about 30% of requests to /profile started returning "field not found" errors; the rest work fine. Checking each instance's own version confirms exactly the mid-rollout pattern above — 3 of 10 instances on the new version. 3 out of 10 is 30%, matching the failure rate almost exactly.

The new version's code expects a display_name field that a migration was meant to backfill onto every existing user record — but the backfill is still running and only partially complete. Requests landing on new-version instances fail specifically for the subset of accounts the backfill hasn't reached yet. The fix: pause the rollout, let the backfill finish fully, confirm the field is populated everywhere, then resume — the expand-contract discipline this chapter names, applied correctly this time by fixing the sequencing rather than rolling forward blind.

Hands-On Exercises