Diagnosing "It's Returning the Wrong Thing": A Second Troubleshooting Walkthrough

Logging & Log Analysis

Chapter 7 · Diagnosing "It's Returning the Wrong Thing": A Second Troubleshooting Walkthrough

Chapter 6 was one thread of evidence, layered until it pointed at a single cause. This chapter is different on purpose: four genuinely plausible causes, each checked and eliminated with real evidence, until only one survives. "Wrong response" problems are rarely obvious from the symptom alone — this is the second half of the user's own original example, worked through directly.

The Ticket

"The product page shows different prices for different users — some see today's updated prices, some still see yesterday's. Started right after this morning's price update went out."

Four candidate explanations are worth considering before touching anything: a bug in the update itself, a caching layer serving stale content, a misconfigured virtual host serving from the wrong source, or a backend that didn't receive the update. The temptation is to jump straight to the one that sounds most familiar — here, that's caching, since "different users see different things" is the classic caching fingerprint. This walkthrough deliberately doesn't skip the other three just because one guess feels obviously right.

Step 1 — Was the update actually applied correctly?Ruled out

Checking the application's own log for this morning's price update confirms it completed successfully, updating every affected product:

[08/Aug/2026:07:00:04] [INFO] Price update job completed: 1,204 products updated, 0 failures

The source of truth is correct. Whatever's serving stale prices, it isn't because the update itself failed or ran incompletely.

Step 2 — Is a caching layer serving stale content?Ruled out

This is the tempting answer, so it gets checked properly rather than assumed. A quick request with response headers visible shows no cache layer is even in front of this path:

curl -I https://example.com/product/4821 HTTP/1.1 200 OK X-Cache: (header not present at all)

No X-Cache, no Age header, nothing indicating a cache sits in front of this request. The symptom looked like caching, but there's no cache configured here to actually be the cause.

Step 3 — Is the wrong virtual host serving some requests?Ruled out

Checking the access log's own recorded Host field for both an "old price" and a "new price" request confirms both landed on the same, correct virtual host, serving from the same document root:

[08/Aug/2026:09:14:02] Host: shop.example.com 200 # showed old price [08/Aug/2026:09:14:19] Host: shop.example.com 200 # showed new price

Identical host, identical config on both requests. Whatever's different between these two, it isn't which virtual host answered.

Step 4 — Did every backend actually receive the deployment?Confirmed

This site runs behind an Nginx reverse proxy, load-balancing across two Apache backends — a detail worth checking directly, using an Nginx log format that includes $upstream_addr, the actual backend server that handled each request:

[08/Aug/2026:09:14:02] upstream: 10.0.0.11:80 # backend-1 — showed old price [08/Aug/2026:09:14:19] upstream: 10.0.0.12:80 # backend-2 — showed new price

Requests alternate between two backend servers, and only one of them is actually serving new prices. The rollout of this morning's deployment reached backend-2 but never completed on backend-1 — which explains the exact symptom precisely: which price a user sees depends entirely on which backend the load balancer happens to route them to on that request.

The right-sounding answer isn't the same as the confirmed answer
Caching was a genuinely reasonable first guess — the symptom pattern really does match how caching problems usually look. The discipline that mattered here wasn't avoiding that guess, it was refusing to stop at it without checking. A pattern match is a starting hypothesis, not a diagnosis, until something in the logs actually confirms it.

Hands-On Exercises

Exercise 1

Explain what specifically ruled out caching as the cause in Step 2, and why "different users see different things" wasn't enough evidence on its own to confirm it.

📄 View solution
Exercise 2

Explain what $upstream_addr revealed in Step 4, and why this piece of information required a specific log configuration choice rather than being present in a default Nginx access log.

📄 View solution
Exercise 3

Explain why this chapter says caching being "a genuinely reasonable first guess" wasn't the actual problem with jumping to it — what was the real mistake this chapter warns against?

📄 View solution

Chapter 7 Quick Reference

  • "Wrong response" problems often have several plausible causes — check and eliminate each one with real evidence, don't stop at whichever guess feels most familiar
  • Confirm the update actually succeeded at the source before assuming it's a delivery/serving problem
  • Caching is ruled in or out by response headers (X-Cache, Age) — not by symptom pattern-matching alone
  • A misconfigured virtual host is ruled out by confirming the Host field and served content genuinely match across requests
  • In a load-balanced setup, which backend actually served a request ($upstream_addr in Nginx) can reveal a partially-completed deployment — a genuinely different cause from all three of the others
  • Next chapter: Log Aggregation & Centralized Logging