Health Checks & Self-Healing

Chapter 5's self-healing and Chapter 2's reconciliation loop both assume Kubernetes can actually detect a problem. This chapter covers exactly how.

The Problem — "Running" Doesn't Mean "Healthy"

Revisiting Chapter 3's pod phase material: a container can sit in the "Running" phase while being completely unresponsive, deadlocked, or stuck in an infinite loop. The process technically hasn't crashed, so Kubernetes' default behavior — restarting only on an actual process exit — wouldn't catch this at all. "The process is running" and "the application is working correctly" are not the same claim.

Probes — How Kubernetes Actually Checks

• Liveness — "is this container still alive and functioning, or should it be restarted?"
• Readiness — "is this container ready to receive traffic right now?" (directly tied to Chapter 6's Service/Endpoints material.)
• Startup — "has this container finished its possibly slow startup process yet?" — used to prevent liveness probes from prematurely killing a container still legitimately starting up.

Liveness Probes — Restart on Failure

A failed liveness probe causes Kubernetes to kill and restart the container. This is what actually triggers self-healing at the container level — genuinely distinct from Chapter 5's pod-replacement-level self-healing via the ReplicaSet. A liveness probe failure restarts a container within the same pod; a pod being deleted entirely is a separate, higher-level event handled by the ReplicaSet.

Readiness Probes — Removing From Service Without Killing

Sometimes a container is perfectly alive and functioning, but temporarily not ready to serve traffic — still loading a large dataset at startup, or briefly overloaded and needing a moment to catch up. Killing and restarting it in this case would be actively counterproductive. A readiness probe instead just temporarily removes the pod from the Service's Endpoints (Chapter 6) until it reports ready again — no restart involved at all.

Startup Probes — Handling Slow-Starting Applications

Some applications — heavy initialization work like loading a large ML model or warming a cache — take much longer to become ready than a typical liveness probe's failure threshold tolerates, causing the liveness probe to repeatedly kill the container before it ever finishes starting. A genuinely frustrating, common real problem. A startup probe defers liveness (and readiness) probing entirely until it itself succeeds once, giving slow-starting applications the room they genuinely need.

Probe Mechanics — HTTP, TCP & Exec

• HTTP GET — the most common; checks for a successful response code from a specified path.
• TCP socket — simply checks whether a port accepts a connection; useful for non-HTTP services.
• Exec — runs a command inside the container and checks its exit code; most flexible, also the most expensive and slowest of the three.

Configurable timing parameters — initialDelaySeconds, periodSeconds, failureThreshold — are genuinely practical knobs worth knowing exist, even without exhaustive detail on each here.

A Concrete Example — Liveness & Readiness Together

A genuinely common real-world pattern: an application exposes two separate health endpoints with distinct meanings, matching the liveness-vs-readiness distinction explained above.

Restart Policies

spec.restartPolicy controls what happens when a container exits: Always (default, appropriate for long-running services like Deployments), OnFailure (only restarts on a non-zero exit — relevant for Course 2's own Jobs material), Never. This is a pod-level setting, genuinely distinct from the per-container liveness-probe-triggered restarts covered above — two related but different restart mechanisms worth not conflating.

How This All Ties Together — The Full Self-Healing Picture

The reconciliation loop (Chapter 2) is the general mechanism. ReplicaSets (Chapter 5) use it to maintain pod count. Liveness probes (this chapter) tell Kubernetes when a specific container needs restarting. Readiness probes (this chapter) tell Services (Chapter 6) which pods should currently receive traffic. Together, this is what makes Kubernetes self-healing in the full, practical sense Chapter 1 promised — a satisfying convergence of nearly everything this course has covered so far, ahead of Chapter 12's own capstone.

Hands-On Exercises