Capstone — Triaging Three Real Application Tickets

Load time scales with a specific count (organization size), not with overall traffic or time of day — exactly the signature Chapter 6 taught to recognize before even opening a query log.

Query logging confirms it directly: one query for the organization's user list, followed by one additional query per user to fetch that user's role and permission set — 340 total queries for an organization with 339 users. Replacing the per-user loop with a single batched query fetching all roles at once cuts the page load from 12 seconds to well under a second.

Sudden onset, tied to today's deploy, some requests failing rather than all — the error body confirms a familiar shape: {"error": "pool_timeout", "request_id": "..."}, with a correlation ID ready for deeper investigation.

An early guess — a partner integration retrying aggressively and exhausting shared resources — is checked directly via per-key rate-limit consumption. Every key shows normal, unremarkable usage. Ruled out; the real cause lies elsewhere.

Checking each instance's own version confirms the failures cluster specifically on instances already running this morning's release — a genuine version-skew signature, narrowing the search to what actually changed in the new code.

Not simply a slow query this time — connections stuck in PostgreSQL's own idle in transaction state, meaning a transaction was opened and never committed or rolled back. This morning's new code path opens a transaction, but under one specific error condition throws before ever reaching the commit — silently leaking the connection forever, on every instance running the new version. The fix: wrap the transaction in a proper try/finally so it's always closed regardless of how the request ends, shipped as an immediate hotfix.

An early guess — new instances stampeding a cold cache — is checked directly: TTLs are properly jittered, and there's no clock-aligned periodicity in the load pattern. Ruled out.

The session-loss complaint matches this course's own established pattern exactly: the load balancer redistributed sticky-session assignments when the instance pool changed size, and the application still relies on local, per-server session storage rather than a shared store.

Separately, the orchestrator's own event log shows the new instances repeatedly cycling in and out of ready state — flapping, with no actual crashes. Their readiness check includes a call to a third-party shipping-rates API that's been running unusually slowly today, and a slow (not down) dependency is enough to fail the check. The application itself is otherwise perfectly capable of serving most requests; readiness was checking something it shouldn't have owned.