Caching Strategies

Distributed Systems & Scalability

Chapter 3 · Caching Strategies

Caching trades correctness risk for speed — every strategy in this chapter is a different way of drawing that trade. The famous line about cache invalidation being one of the two hard problems in computer science isn't a joke about difficulty in the abstract; this chapter builds a real, verified case of exactly what goes wrong.

Cache-Aside: Check the Cache First, Populate on Miss

class CacheAsideRepository:
    def __init__(self, database): self.database = database; self.cache = {}
    def get(self, key):
        if key in self.cache:
            return self.cache[key]
        value = self.database.get(key)
        self.cache[key] = value
        return value
    

Verified directly — a real, measured speedup on every read after the first

Against a simulated database with real I/O latency, the first read of PROD-1 (a genuine cache miss) took 10.37ms and incremented db.query_count to 1. Twenty subsequent reads of the identical key averaged 0.0005ms each — db.query_count never moved past 1. Cache-hit reads were roughly 19,566× faster than the original cache-miss read.

Write-Through vs. Write-Behind: Where the Real Tradeoff Lives

class WriteThroughCache:
    def set(self, key, value):
        self.cache[key] = value
        self.database.set(key, value)  # synchronous, immediate

class WriteBehindCache:
    def set(self, key, value):
        self.cache[key] = value
        self.pending_writes[key] = value  # NOT written to the database yet
    def flush(self):
        for key, value in self.pending_writes.items():
            self.database.set(key, value)
        self.pending_writes.clear()
    

Verified directly — write-behind is dramatically faster, and that speed is exactly what makes it risky

Twenty writes through write-through took 206.47ms total (each one blocking on the real database write). The identical twenty writes through write-behind, before flush() ever runs, took 0.02ms total — roughly 12,145× faster, because nothing has touched the database yet.

Verified directly — a simulated crash before flush() loses the write entirely

Calling write_behind.set('PROD-1', 100) and then checking database.get('PROD-1') before flush() runs returns None — the write genuinely never reached the database. cache.cache['PROD-1'] correctly shows 100 the whole time. Simulating a crash at exactly this point (the process ends, flush() never gets called) confirms the write is gone — not delayed, gone. Only once flush() actually runs does database.get('PROD-1') correctly return 100.

The tradeoff, stated precisely

Write-through pays the full write cost every time, in exchange for a guarantee: the moment set() returns, the database and cache genuinely agree. Write-behind defers that cost — verified 12,145× faster — but during the deferral window, the only copy of the truth lives in memory, verified vanishing entirely on a simulated crash. This is the identical shape of tradeoff Distributed Systems & Scalability's own future CAP Theorem chapter formalizes: speed and immediate durability aren't both free at the same time.

Cache Invalidation: The Genuinely Hard Part

Reusing Software Architecture Fundamentals' own PricingEngine (Ch.7): what happens when a cached price outlives the data it was calculated from?

Verified directly — a customer sees a real, wrong price after the underlying data genuinely changed

With PROD-1 well-stocked (50 units), CachedPricingService.get_price() correctly caches and returns 100 (no scarcity pricing). Stock then genuinely drops to 5 — a real change that should trigger scarcity pricing. Querying the identical cache key again, without invalidating anything, still returns the stale 100 — while the correct price, computed fresh, is now 115.0. The customer would be quoted the wrong price, with no error, no warning, and nothing in the code path that looks broken.

Verified directly — explicit invalidation is what actually fixes it

Calling cached_service.invalidate('PROD-1') — clearing every cached entry for that product — and querying again correctly returns 115.0, matching the true current price exactly.

Why this is the "hard" part, specifically

Caching itself (Cache-Aside above) was mechanical — check, miss, populate, return. Knowing when a cached value has become wrong requires knowing about every possible change to the data it depends on, from every part of the system that could make one — here, specifically, that a stock update needs to trigger a price-cache invalidation, a connection that isn't visible anywhere in the caching code itself. Missing even one such trigger reproduces the exact bug just verified: a technically-working cache silently serving wrong answers.

CDN Basics

A Content Delivery Network applies cache-aside's own idea geographically: static content (images, scripts, stylesheets) gets cached at servers physically close to each visitor, instead of every request crossing however much distance separates the visitor from the origin server. Software Architecture Fundamentals Chapter 4 measured a real, non-zero cost for crossing a network boundary even on localhost — a CDN exists specifically to shrink that same kind of cost when the distance is a real geographic one, not a loopback address.

Where This Connects

This chapter's finding	What it connects to
Write-behind's verified 12,145× speed gain, traded for a verified real data-loss window	Chapter 5's own CAP Theorem & Consistency Models — the same speed-vs-guarantee trade, formalized
A real, verified stale-price bug from a missing invalidation trigger	Software Architecture Fundamentals Chapter 6's own eventual-consistency finding — both are the same underlying risk (data that's technically present but no longer true) surfacing in different layers
Cache-aside's own 19,566× read speedup	Software Architecture Fundamentals Chapter 7's own ~18,111× testability speedup — a strikingly similar-shaped number, from the identical underlying idea: avoid real, slow I/O whenever it's safe to

Hands-On Exercises

Exercise 1

Extend this chapter's own CacheAsideRepository with a second key, 'PROD-2', and verify that a cache hit on 'PROD-1' doesn't accidentally also count as a hit for 'PROD-2' — confirm the first read of 'PROD-2' still triggers a genuine database query even after 'PROD-1' is fully cached.

📄 View solution

Exercise 2

Using this chapter's own WriteBehindCache, write two values for two different keys before ever calling flush(), then simulate a crash. Verify both writes are lost from the database, and verify calling flush() afterward (simulating a recovery that never happened) can't bring back data that was never in pending_writes to begin with.

📄 View solution

Exercise 3

This chapter's own CachedPricingService.invalidate() requires something else in the system to remember to call it whenever stock changes. Using this chapter's own verified stale-price bug, explain specifically what would have to change elsewhere in a real system (not in the cache itself) to make that invalidation call actually happen reliably.

📄 View solution

Chapter 3 Quick Reference

Cache-aside: check cache, miss falls through to the database and populates the cache — verified: ~19,566× faster on repeated reads, with db.query_count never incrementing past the first miss
Write-through: writes hit cache and database together, synchronously — always consistent, always pays the full write cost
Write-behind: writes hit cache immediately, database later — verified ~12,145× faster, verified genuinely losing data on a simulated crash before flush()
Invalidation, verified as the hard part: a real, wrong price (100 instead of the correct 115.0) was served with zero errors until the cache was explicitly told to forget — the caching mechanism itself was never the problem
Next chapter: Database Scaling — replication, read replicas, and sharding