Health Checks, Readiness Probes & Graceful Shutdown

Web & Application Troubleshooting

Chapter 8 · Health Checks, Readiness Probes & Graceful Shutdown

Chapter 7 covered what goes wrong during a rollout. This chapter covers the machinery that's supposed to make a rollout — or any routine restart — safe in the first place: liveness and readiness checks, and what happens when an instance is shut down without giving it a chance to finish what it was doing.

Liveness vs. Readiness: A Real, Important Distinction

Two genuinely different questions, with genuinely different consequences when the answer is no:

Check	Question it answers	What happens on failure
Liveness	Is this process alive at all?	The orchestrator restarts the container
Readiness	Can this instance currently serve traffic correctly?	The instance is pulled from the load balancer's rotation — not restarted

Using the same check for both is a real, common mistake

A single shared check that's too broad — for example, one that verifies a downstream third-party dependency is reachable — can turn an unrelated outage into needless restarts. A temporary blip in a payment provider's API should mean "temporarily stop sending this instance new requests" (a readiness concern), not "kill and restart this perfectly healthy process" (a liveness consequence) — but with a shared check, the orchestrator can't tell the two apart.

# Separate, correctly-scoped checks
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 2

What a Readiness Probe Should (and Shouldn't) Check

Reasonable readiness checks: can this instance reach its own database connection pool, is its cache connection alive, has startup initialization finished. Unreasonable: checking something unrelated to whether this specific instance can serve traffic — a third-party API's own health, for instance, can take an entire fleet out of rotation over someone else's outage, even though the application itself is otherwise perfectly capable of handling most requests.

The Symptom of a Readiness Probe Gone Wrong

A misconfigured readiness check produces a distinctive pattern: instances repeatedly pulled from rotation and put back — "flapping" — visible as capacity periodically dropping even though nothing actually crashed. Checking the orchestrator's own event history (or a load balancer's own health-check log) directly shows readiness failures, rather than requiring you to assume instances are genuinely down.

Graceful Shutdown: The Other Half of the Lifecycle

When an instance is being replaced — during a deployment, a scale-down, or a routine restart — simply killing the process immediately can cut off requests that were still mid-processing. A well-behaved shutdown sequence does three things in order: mark the instance not-ready (so the load balancer stops sending new requests), wait for in-flight requests to actually finish (a drain period), and only then terminate.

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 15"]
terminationGracePeriodSeconds: 30

Skipping the drain step produces its own recognizable symptom: a small, brief burst of connection-reset or aborted-request errors, correlating precisely with deployment or scale-down events.

Distinct from Chapter 7's own deploy-correlated symptom

Both are deploy-correlated, but they're not the same problem: version skew (Chapter 7) produces wrong data or contract mismatches, because two genuinely different code versions are both answering requests. A missing drain period produces dropped or reset connections specifically at the exact moment of termination, because a request was still in flight when its server was killed out from under it. Telling them apart matters — the fixes are completely different.

A Concrete Symptom-to-Cause Table

Symptom	Likely cause
Capacity flaps up and down, no actual crashes	Readiness check misconfigured — too strict, or checking the wrong thing
Brief burst of connection resets, exactly at deploy/scale-down moments	No graceful shutdown/drain period configured
Unnecessary restarts correlating with an unrelated dependency's own outage	Liveness check too broad — checking something readiness should own instead

Working Example: The 20-Request Deploy Blip

A fresh ticket: every deployment causes a brief spike of roughly 20 failed "connection reset" requests, lasting just a few seconds — the deployment itself completes successfully, and the new version works fine immediately afterward. Checking the deployment process confirms old instances are terminated the instant the new version becomes ready, with no drain period configured at all. Requests still in flight on an old instance at that exact moment get abruptly cut off mid-response. Adding a preStop hook that pauses before actual termination — giving the load balancer time to stop routing new traffic and letting existing requests finish — resolves the blip entirely without changing anything about the deployment's own speed or correctness.

Hands-On Exercises

Exercise 1

Explain why using the same check for both liveness and readiness is a real mistake, using this chapter's own payment-provider example.

📄 View solution

Exercise 2

Explain the difference between the symptom caused by version skew (Chapter 7) and the symptom caused by a missing drain period, and why they need different fixes.

📄 View solution

Exercise 3

In this chapter's worked example, explain exactly why in-flight requests were being cut off, and how a preStop hook fixes it without changing the deployment's own logic.

📄 View solution

Chapter 8 Quick Reference

Liveness = is the process alive (failure → restart); readiness = can it serve traffic right now (failure → pulled from rotation, not restarted)
A shared check for both is a real mistake — an unrelated dependency blip can trigger needless restarts
Readiness should check this instance's own ability to serve — not a third-party dependency's own health
A flapping capacity count with no real crashes = a misconfigured readiness check
Graceful shutdown: mark not-ready, drain in-flight requests, then terminate — skipping it causes connection resets exactly at deploy/scale-down moments
Distinct from version skew: wrong data/contract mismatch (Ch7) vs. dropped connections at the moment of termination (this chapter)
Next chapter: Rate Limiting & Throttling Symptoms