"The checkout API is throwing 500 errors, intermittently, during busy periods." One engineer's first instinct is to check the network and the server's own resources — reasonable instincts, and exactly what Network Troubleshooting and System Monitoring & Performance Diagnosis teach. The other engineer checks those too, finds them clean, and knows there's a whole category of genuine causes that live one layer higher — specific to running an actual application, not the machine or network underneath it. This course is entirely about that layer.

What the Other Three Courses Already Cover

This course assumes those three are either already ruled out or being checked in parallel — it doesn't re-teach any of them. What it covers instead is the set of things that can genuinely go wrong even when the network is clean, resources are healthy, and the logs don't show an obvious smoking gun.

The Genuine Gap: What Lives Above the Resource Layer

A short preview of what the rest of this course actually covers — all of it specific to running a web application, none of it explained by network or raw resource exhaustion: database connection pool exhaustion, caching bugs (stale data and cache stampedes), session and state issues in load-balanced environments, slow queries and N+1 patterns, deployment-related version skew, health checks and graceful shutdown, and rate limiting.

Nothing about this error involves the network, the CPU, or the disk — the machine is fine. You'll learn to recognize and resolve exactly this pattern in Chapter 3.

A Concrete Example: The Same "500 Error," Six Different Layers

"The API returned a 500" is one symptom that genuinely could be caused by something in any of this subject's four courses — worth seeing side by side, so this course's own boundary is clear from the start:

Six genuinely different root causes, the exact same reported symptom. Knowing which of the four courses actually owns a given cause is most of the battle — this table is worth remembering as later chapters build out the three "this course" rows in full.

The Shared Discipline Still Applies

Even though the content here is genuinely new, the method carrying it isn't: scope the complaint before naming a cause, gather real evidence rather than guessing, and don't take an action (a restart, a redeploy) that destroys the evidence needed to actually understand what happened — the same principles log1, netdiag1, and perfdiag1 each opened with in their own first chapters, still doing the same work here.

What This Course Covers

Reading the 5xx family as a genuine diagnostic language, database connection pool exhaustion, caching bugs, session/state issues in load-balanced setups, slow queries and N+1 patterns, deployment-related version skew, health checks and graceful shutdown, and rate limiting. The capstone applies all of it to three realistic application tickets.

Hands-On Exercises

Network Troubleshooting's own Chapter 9 already covered what 500, 502, 503, and 504 mean from a reverse proxy's point of view — a genuine, valuable finding: any HTTP response at all, even an error, proves DNS, the network path, the port, and TLS all worked. This chapter starts exactly where that one stopped: once you know it's a genuine application-layer 500 (not a proxy failing to reach its backend at all), what does the application's own response actually tell you?

Recap: The Proxy-Level View

That table answers "did the request reach the application, and did something come back." This chapter is about the 500s that do reach the application — and specifically, what the application chose to say about it.

500 Isn't One Thing: Reading the Error Body

A well-built API rarely returns an empty 500 — it returns a structured error body, and that body is often far more informative than the bare status code. Worth being honest about a genuine, common wrinkle first:

The Correlation ID: A Direct Line to the Right Log Entry

Well-designed APIs return a request ID or correlation ID with every response — in the body, a header like X-Request-Id, or both — specifically so a user-facing error can be matched to the exact server-side log line that explains it. This is a genuine upgrade over Logging & Log Analysis's own timestamp-correlation technique: instead of narrowing down logs by "roughly when this happened," a correlation ID lets you grep for the exact request, no ambiguity at all.

All Requests Failing vs. Some Requests Failing

A genuinely useful distinguishing question before reading a single error body: what fraction of requests to this endpoint are actually failing?

This single distinction alone points you toward roughly half of this course's own remaining chapters before you've read a single log line.

Working Example: Reading Chapter 1's Own Ticket

Chapter 1 left an open ticket: the checkout API returning intermittent 500s under load, with network and resources already ruled out. This chapter's own techniques resolve it immediately — the error body itself:

No log-grepping needed — the application is telling us directly: this is a connection pool timeout, it's expected to be transient ("retryable": true), and a correlation ID is available if deeper investigation is still needed. Combined with the "some requests, not all" failure pattern from earlier in this chapter, this confirms the exact category Chapter 1 previewed. Chapter 3 covers connection pool exhaustion in full — reading this exact ticket through to its actual fix.

Hands-On Exercises

Chapter 2 fully identified the checkout ticket's error: pool_timeout, retryable, correlation ID in hand. This chapter is about actually resolving it — connection pools, why they exist, the two genuinely different reasons one runs dry, and why the tempting quick fix can make the real cause worse.

What a Connection Pool Is and Why It Exists

Opening a new database connection is expensive — a TCP handshake, authentication, session setup — expensive enough that doing it fresh for every single request would be far too slow. A connection pool solves this by keeping a fixed set of connections already open, which application code borrows for the duration of a query and returns immediately afterward, ready for the next request to reuse.

Reading Pool Metrics

Active connections are currently checked out and in use; idle ones are open but available; waiting is the number of requests currently blocked, unable to get a connection at all — exactly this course's own opening example from Chapter 1. All 20 connections in use, none idle, 14 more requests queued behind them: the pool is genuinely exhausted. The interesting question is why.

Two Genuinely Different Root Causes

This is genuinely the same shape of question System Monitoring & Performance Diagnosis's own Chapter 9 asked about a rising resource trend — is this real growth, or a leak — just applied to a connection pool instead of memory.

Telling the Two Apart: Connection Hold Time

The active/idle/waiting counts alone can't distinguish the two — both look identical from that view alone. What actually separates them is how long connections are being held. If most connections check out and return quickly, and the pool is simply saturated by genuinely high concurrent traffic, that's a capacity problem. If a small handful of connections are held for seconds or minutes while everything else churns normally in milliseconds, that's a small number of culprits starving the whole pool — not a sizing problem at all.

Finding the Specific Culprit

Most databases expose exactly this — which queries are currently running, and for how long:

One query has been running for over four minutes — dramatically longer than everything else on the list, which is finishing in a couple of seconds. This isn't a capacity problem; it's one specific query holding a connection hostage while everything else is genuinely fine.

Working Example: Fully Resolving the Checkout Ticket

Chapter 1's ticket, finally closed out: pg_stat_activity confirms exactly one query — a full scan of order_items with no supporting index on the column it filters by — has been running for over four minutes, while every other query on the system finishes normally. That single stuck query is holding a connection the whole time, and under peak checkout traffic, enough concurrent requests hit the same slow code path to exhaust the pool entirely, producing the "some requests fail, not all" pattern Chapter 2 identified. The actual fix is adding the missing index (or rewriting the query) — not increasing the pool size, which would only mask the same problem a little longer while adding more load to an already-struggling database.

Hands-On Exercises

A cache trades correctness for speed — it serves a stored copy of data instead of recomputing or refetching it, on the bet that the copy is still good enough to use. That bet is usually right. This chapter is about the two genuinely different ways it goes wrong: a cache quietly serving something that's no longer true, and a cache emptying itself all at once and taking the backend down with it for a few seconds.

Cache Invalidation: The Genuinely Hard Problem

Two common strategies, each with a real, honest tradeoff:

Neither approach is simply "better" — TTL-based caching accepts a known, bounded staleness window up front; explicit invalidation aims for zero staleness but is only as reliable as every single write path that's supposed to trigger it, which in a real system with multiple services and admin tooling is genuinely easy to miss.

Reading a Stale-Data Complaint

Before assuming a "wrong data" complaint is a cache bug, confirm it directly — compare what the cache is actually returning against what the real source of truth currently holds for the same key:

A genuine mismatch confirms a stale cache. If the two actually agree, the cache isn't the problem at all — the "wrong data" complaint is something else entirely (a display bug, a genuine data error, or user confusion), and continuing to chase caching as the cause would be a wasted detour.

Cache Stampede: When Many Requests Miss at Once

A stampede (also called a thundering herd) happens when a popular cache key expires, and many concurrent requests all miss the cache at the exact same instant — all of them then hit the backend simultaneously to regenerate the same value, producing a sudden, sharp spike in backend load precisely at the moment of expiry.

Real mitigations worth knowing: jittered TTLs (randomizing expiry slightly per key, so not every instance expires at the exact same moment), a single-flight pattern (only one request regenerates the value while everyone else waits for that result instead of duplicating the work), and stale-while-revalidate (serving the slightly-stale value immediately while quietly regenerating it in the background).

Recognizing a Stampede From the Outside

A stampede caused by a fixed TTL has a genuinely distinctive signature: it recurs at a suspiciously regular, clock-aligned interval — every hour, on the hour, if the TTL is exactly 3600 seconds. That periodicity itself is real diagnostic evidence, not a coincidence worth ignoring.

Working Example: The Hourly Database Spike

A fresh ticket: every hour, right on the hour, the database briefly spikes to near-100% CPU for about 10 seconds, then returns to normal — otherwise, everything is healthy. The clock-aligned periodicity is the immediate tell. Checking a suspected key's own remaining time-to-live confirms it:

A TTL close to a full hour, on a genuinely popular key — the homepage's own featured-products list, requested by nearly every visitor. Every instance of the application shares the same cache key, so every one of them experiences the exact same expiry moment simultaneously, and every request that arrives in that brief window misses the cache and hits the database at once. Adding a small amount of random jitter to the TTL (so different instances' copies expire at slightly different times) spreads that load out instead of concentrating it into one sharp spike every hour.

Hands-On Exercises

Spread traffic across several application servers behind a load balancer, and a real question appears that a single-server setup never has to ask: where does a logged-in user's session actually live? If each server only keeps sessions in its own local memory, a session exists on exactly one server — whichever one happened to handle the login. This chapter is about what happens when a later request lands somewhere else.

Sticky Sessions (Session Affinity)

One fix: configure the load balancer to route every request from a given user consistently to the same backend server, usually via a cookie identifying which server they were first assigned to. It solves the local-memory problem without needing any shared infrastructure — but it has a genuine, honest failure mode of its own.

Shared Session Stores

The more resilient alternative: store session data in a shared, external store — commonly Redis — that every application server can read from and write to, regardless of which one a given request lands on. Any server can now correctly serve any user, removing the single-server session-loss risk sticky sessions carry.

Recognizing the Symptom Pattern

"Logged out randomly," "my cart emptied," or "had to log in again mid-session" — reported by some users but not others, especially clustering around a recent deployment or scaling event — is the classic tell. Worth being precise about how this differs from Chapters 3 and 4's own "some requests fail" patterns: there, the "some" correlated with timing and resource contention. Here, the "some" correlates with which server a user happens to be routed to — a genuinely different kind of "intermittent," worth telling apart before assuming the same category of cause applies again.

A Practical Diagnostic Check

Two things worth checking directly: whether the timing lines up with a deployment or scaling event (the same "does it correlate with a known event" question from Chapter 1, applied here), and whether the sticky-session cookie is actually present and being honored.

A missing or unexpectedly absent affinity cookie — stripped by a misconfigured proxy or CDN somewhere in the path, or a load balancer configuration change — would explain session loss just as directly as a server restart would.

Working Example: The Scaling Event That Reshuffled Sessions

A fresh ticket: since yesterday's autoscaling event added two new server instances, roughly 15% of users report being logged out mid-session, seemingly at random. Investigating confirms sticky sessions are configured — but the application was never using a shared session store, only local, per-server memory. Users who happened to remain assigned to the original server instances continued fine. But many load balancers redistribute existing connections across the new, larger pool of backends when the pool itself changes — exactly what happened here, reshuffling a portion of users onto instances that had never seen their session before. Their local-only session simply didn't exist on the new server, and they were silently logged out.

The fix isn't reverting the scaling event — it's migrating to a shared, Redis-backed session store, so a scaling event (or any future one) no longer has the power to erase sessions just by changing which server happens to answer a given request.

Hands-On Exercises

Logging & Log Analysis's own capstone found an N+1 pattern by comparing query counts before and after a deployment. This chapter goes deeper: how to recognize an N+1 pattern directly, how to tell it apart from a single genuinely slow query (a different problem with a completely different fix), and how to read just enough of an EXPLAIN plan to spot the single most common cause of a slow query without needing a full database-tuning background.

What N+1 Actually Is

Instead of one query that fetches everything needed at once — a JOIN, or a single query with an IN clause — the code runs one query to get a list of N items, then loops through them and runs N additional individual queries, one per item, to fetch each one's related data. N+1 total queries where 1 or 2 would have done the job. A very common ORM pitfall, usually from a relationship being lazily loaded inside a loop without anyone noticing.

Recognizing N+1 vs. a Single Slow Query

The fix for each is completely different, so telling them apart matters — and the distinguishing evidence is straightforward: count the queries a single request triggers, and look at each one's own duration.

One long query and dozens of individually-fast ones can add up to almost the same total time — but the fix couldn't be more different: optimizing or indexing one query, versus restructuring the code so it stops looping and issuing a query per item.

Reading EXPLAIN for a Genuinely Slow Single Query

A full database-tuning background isn't needed to catch the single most common cause of a slow query — a full table scan on a large table:

Seq Scan means the database read through the entire table — 2.5 million rows, to find one matching row — because no index exists on the column being filtered. Contrast with a healthy result on an indexed column:

Same query shape, over 50,000 times faster — the entire difference is having (or not having) the right index. Recognizing Seq Scan on a large table in an EXPLAIN output is, by itself, one of the highest-value diagnostic skills in this chapter.

The N+1 Fix, at a Practical Level

Two standard fix shapes worth being able to describe clearly, even if the actual code change belongs to a developer: batch loading (one additional query using an IN clause to fetch every needed related record at once, instead of one query per item) or a JOIN that retrieves everything in a single query from the start.

Working Example: The Slow Order History Page

A fresh ticket: the order history page takes 8+ seconds to load for users with a long order history, but loads instantly for new users with few orders. That user-count correlation is itself a strong early hint — a single slow query wouldn't care how many orders a particular user has; an N+1 pattern would scale exactly this way. Query logging confirms it directly: one query fetching the order list, followed by one additional query per order to fetch that order's line items — 51 total queries for a user with 50 orders, each individually fast, together adding up to the full 8 seconds plus per-query round-trip overhead. Replacing the per-order loop with a single batched query (an IN clause covering every order ID from the first query) cuts the page load from 8 seconds to under 200ms — the same total data, retrieved in two queries instead of fifty-one.

Hands-On Exercises

Chapter 2 named "100% of requests fail, suddenly" as the pattern most likely to point at a deployment. This chapter gives that heuristic its fullest treatment — what actually happens during the window a rolling deployment is in progress, why a database migration can fail independently of the code deploy that depends on it, and a concrete technique for confirming a deployment is the cause rather than just suspecting it.

Version Skew: Old and New Code, Running at the Same Time

A rolling or blue-green deployment inevitably passes through a window where old and new code run simultaneously, across different server instances. If the new version changes an API contract in a way that isn't backward- or forward-compatible — a renamed field, a newly required field, a changed response shape — requests can fail unpredictably depending on which specific instance happens to handle them during that window. This applies to service-to-service communication too, not just client requests: a frontend expecting the new shape can just as easily hit an old backend instance, or the reverse.

This directly explains a genuinely confusing symptom: "it works sometimes and fails other times, right after a deploy, with no code changes since" — because which instance answers a given request is effectively random during the rollout, and each instance is running one version or the other, never a blend.

Recognizing Version Skew From the Outside

Directly checking each instance's own reported version settles this immediately:

Three of ten instances already on the new version, seven still on the old — a rollout genuinely mid-flight. In a container-orchestrated environment, checking each running pod's own image tag serves the same purpose.

Database Migration Failures

A schema migration can fail partway through, or succeed on the database while the application code deployed alongside it doesn't actually match what the migration has finished doing yet. A genuinely common real mistake: deploying code that expects a new field before the migration that backfills it has actually completed everywhere — or the reverse, old code still running against a schema that's already been changed underneath it.

A Genuine Technique: Bisecting by Time

When an error rate shows a sudden, sharp step-change at one specific timestamp — not a gradual climb — checking exactly what deployed or migrated at that precise moment is almost always faster than examining application code in the abstract. The deployment history and migration log are themselves evidence, in exactly the same spirit as this subject's own "check first, don't guess" theme.

Working Example: The 30% "Field Not Found" Ticket

A fresh ticket: right after this afternoon's deploy, about 30% of requests to /profile started returning "field not found" errors; the rest work fine. Checking each instance's own version confirms exactly the mid-rollout pattern above — 3 of 10 instances on the new version. 3 out of 10 is 30%, matching the failure rate almost exactly.

The new version's code expects a display_name field that a migration was meant to backfill onto every existing user record — but the backfill is still running and only partially complete. Requests landing on new-version instances fail specifically for the subset of accounts the backfill hasn't reached yet. The fix: pause the rollout, let the backfill finish fully, confirm the field is populated everywhere, then resume — the expand-contract discipline this chapter names, applied correctly this time by fixing the sequencing rather than rolling forward blind.

Hands-On Exercises

Chapter 7 covered what goes wrong during a rollout. This chapter covers the machinery that's supposed to make a rollout — or any routine restart — safe in the first place: liveness and readiness checks, and what happens when an instance is shut down without giving it a chance to finish what it was doing.

Liveness vs. Readiness: A Real, Important Distinction

Two genuinely different questions, with genuinely different consequences when the answer is no:

What a Readiness Probe Should (and Shouldn't) Check

Reasonable readiness checks: can this instance reach its own database connection pool, is its cache connection alive, has startup initialization finished. Unreasonable: checking something unrelated to whether this specific instance can serve traffic — a third-party API's own health, for instance, can take an entire fleet out of rotation over someone else's outage, even though the application itself is otherwise perfectly capable of handling most requests.

The Symptom of a Readiness Probe Gone Wrong

A misconfigured readiness check produces a distinctive pattern: instances repeatedly pulled from rotation and put back — "flapping" — visible as capacity periodically dropping even though nothing actually crashed. Checking the orchestrator's own event history (or a load balancer's own health-check log) directly shows readiness failures, rather than requiring you to assume instances are genuinely down.

Graceful Shutdown: The Other Half of the Lifecycle

When an instance is being replaced — during a deployment, a scale-down, or a routine restart — simply killing the process immediately can cut off requests that were still mid-processing. A well-behaved shutdown sequence does three things in order: mark the instance not-ready (so the load balancer stops sending new requests), wait for in-flight requests to actually finish (a drain period), and only then terminate.

Skipping the drain step produces its own recognizable symptom: a small, brief burst of connection-reset or aborted-request errors, correlating precisely with deployment or scale-down events.

A Concrete Symptom-to-Cause Table

Working Example: The 20-Request Deploy Blip

A fresh ticket: every deployment causes a brief spike of roughly 20 failed "connection reset" requests, lasting just a few seconds — the deployment itself completes successfully, and the new version works fine immediately afterward. Checking the deployment process confirms old instances are terminated the instant the new version becomes ready, with no drain period configured at all. Requests still in flight on an old instance at that exact moment get abruptly cut off mid-response. Adding a preStop hook that pauses before actual termination — giving the load balancer time to stop routing new traffic and letting existing requests finish — resolves the blip entirely without changing anything about the deployment's own speed or correctness.

Hands-On Exercises

A 429 looks alarming in a dashboard full of otherwise-green metrics, but it's a genuinely different kind of signal from everything else this course has covered so far — a rate limit rejecting a request isn't a sign anything is broken. This chapter closes out the course's content chapters by reading that signal correctly, and by recognizing one more version of a pattern that's now appeared several times: aggregate exhaustion caused either by genuine broad demand, or by one specific culprit.

"Throttled" Is Not "Down"

A 429 Too Many Requests is the server explicitly saying: I'm healthy, I received your request, and I'm deliberately declining to process it right now. That's a fundamentally different situation from a 5xx (the server struggling) or a timeout (the network or server genuinely unresponsive) — recognizing a 429 immediately for what it is avoids wasting time chasing a "the service is down" theory when the service is, in fact, working exactly as designed.

Reading the Rate Limit Headers

These headers turn a vague "we got throttled" into a precise statement: this client's limit is 1,000 requests per window, 0 remain, and it resets in 42 seconds. No guessing needed about when the client will be unblocked.

Client-Side vs. Server-Side: Who Actually Applied the Limit?

A rate limit might be the application's own internal per-user or per-key limit, protecting its own resources — or it might come from somewhere entirely outside the application's control, like a third-party API the app calls being rate-limited, or a CDN/WAF applying its own limit before a request even reaches the application at all. This is genuinely the same ambiguity Network Troubleshooting's own Chapter 9 raised about a 403 — the response code alone doesn't always say which layer actually produced it.

Two Genuinely Different Symptom Patterns

A pattern this course keeps returning to, in a new shape one more time: aggregate exhaustion caused either by genuine, broad demand exceeding a limit that's simply set too low (Chapter 3's own connection-pool "capacity" case, again) — or by one small number of culprits consuming a disproportionate share of a shared pool (Chapter 3's own "held too long" case, again, this time a client hammering a rate limit rather than a query holding a connection).

Finding the Actual Culprit

Most rate limiters can report consumption broken down by client, API key, or IP — not just the aggregate. A single key consuming a wildly disproportionate share points directly at the second cause:

Working Example: The Partner Integration's Retry Bug

A fresh ticket: since this morning, a significant fraction of API requests from mobile app users are returning 429s — even though overall traffic hasn't meaningfully grown. The per-key breakdown above tells the real story immediately: one specific API key, belonging to a single integration partner rather than typical mobile traffic, accounts for over 87% of requests against the shared limit. A recent bug in that partner's own retry logic started triggering far more often this morning, and — ignoring Retry-After entirely — it retries immediately on every 429, compounding the problem with every failed attempt.

This is genuinely not a capacity problem — the limit was perfectly adequate for real traffic before this partner's bug started firing — and it's not something this application's own team can fix directly, since the broken retry logic lives in someone else's system. The honest resolution here is external: reaching out to the partner to fix their retry behavior, and, in the meantime, applying a tighter limit specifically scoped to that one key so it stops degrading service for every other legitimate client sharing the same overall pool.

Hands-On Exercises

Nine chapters built the pieces — knowing this course's own boundary against its three siblings, reading error bodies, connection pools, caching, sessions, slow queries, deployments, health checks, and rate limits. This capstone applies all of it to three fresh tickets, worked more briskly than earlier chapters' own dedicated walkthroughs, since the underlying discipline should already feel familiar by now.

Ticket 1: "The admin dashboard is unusably slow for our biggest customers"

The admin dashboard's user list page loads instantly for small organizations, but takes 12+ seconds for organizations with hundreds of users.

Ticket 2: "Since this morning's deploy, we're intermittently getting connection pool exhaustion"

Roughly a third of requests to a specific endpoint have started failing with pool timeouts since this morning's release; the rest work fine.

Ticket 3: "Since today's scale-up, users are getting logged out, and our real capacity seems lower than it should be"

Two complaints arriving together right after scaling up for a traffic event: random session loss, and capacity that doesn't match the number of instances actually running.

Two genuinely separate fixes for two genuinely separate causes, both surfaced by the same scaling event: migrating to a shared, Redis-backed session store, and narrowing the readiness check to only cover this instance's own true dependencies.

Chapter Attribution

Honest Scope Note

Hands-On Exercises