Diagnosing "It's Returning the Wrong Thing": A Second Troubleshooting Walkthrough
Logging & Log Analysis
Chapter 7 · Diagnosing "It's Returning the Wrong Thing": A Second Troubleshooting Walkthrough
Chapter 6 was one thread of evidence, layered until it pointed at a single cause. This chapter is different on purpose: four genuinely plausible causes, each checked and eliminated with real evidence, until only one survives. "Wrong response" problems are rarely obvious from the symptom alone — this is the second half of the user's own original example, worked through directly.
The Ticket
"The product page shows different prices for different users — some see today's updated prices, some still see yesterday's. Started right after this morning's price update went out."
Four candidate explanations are worth considering before touching anything: a bug in the update itself, a caching layer serving stale content, a misconfigured virtual host serving from the wrong source, or a backend that didn't receive the update. The temptation is to jump straight to the one that sounds most familiar — here, that's caching, since "different users see different things" is the classic caching fingerprint. This walkthrough deliberately doesn't skip the other three just because one guess feels obviously right.
Checking the application's own log for this morning's price update confirms it completed successfully, updating every affected product:
The source of truth is correct. Whatever's serving stale prices, it isn't because the update itself failed or ran incompletely.
This is the tempting answer, so it gets checked properly rather than assumed. A quick request with response headers visible shows no cache layer is even in front of this path:
No X-Cache, no Age header, nothing indicating a cache sits in front of this request. The symptom looked like caching, but there's no cache configured here to actually be the cause.
Checking the access log's own recorded Host field for both an "old price" and a "new price" request confirms both landed on the same, correct virtual host, serving from the same document root:
Identical host, identical config on both requests. Whatever's different between these two, it isn't which virtual host answered.
This site runs behind an Nginx reverse proxy, load-balancing across two Apache backends — a detail worth checking directly, using an Nginx log format that includes $upstream_addr, the actual backend server that handled each request:
Requests alternate between two backend servers, and only one of them is actually serving new prices. The rollout of this morning's deployment reached backend-2 but never completed on backend-1 — which explains the exact symptom precisely: which price a user sees depends entirely on which backend the load balancer happens to route them to on that request.
Hands-On Exercises
Explain what specifically ruled out caching as the cause in Step 2, and why "different users see different things" wasn't enough evidence on its own to confirm it.
📄 View solutionExplain what $upstream_addr revealed in Step 4, and why this piece of information required a specific log configuration choice rather than being present in a default Nginx access log.
📄 View solutionExplain why this chapter says caching being "a genuinely reasonable first guess" wasn't the actual problem with jumping to it — what was the real mistake this chapter warns against?
📄 View solutionChapter 7 Quick Reference
- "Wrong response" problems often have several plausible causes — check and eliminate each one with real evidence, don't stop at whichever guess feels most familiar
- Confirm the update actually succeeded at the source before assuming it's a delivery/serving problem
- Caching is ruled in or out by response headers (
X-Cache,Age) — not by symptom pattern-matching alone - A misconfigured virtual host is ruled out by confirming the
Hostfield and served content genuinely match across requests - In a load-balanced setup, which backend actually served a request ($upstream_addr in Nginx) can reveal a partially-completed deployment — a genuinely different cause from all three of the others
- Next chapter: Log Aggregation & Centralized Logging