Fault Tolerance & Resilience Patterns

Distributed Systems & Scalability

Chapter 9 · Fault Tolerance & Resilience Patterns

Software Architecture Fundamentals Chapter 4 measured a downstream slowdown cascading straight up through a chain of synchronous calls. Technical Support's own perfdiag1/incident1 courses spend whole chapters diagnosing exactly this kind of failure after the fact. This chapter builds the four patterns that stop it from reaching that point at all — and verifies what each one actually buys.

Circuit Breakers: Fail Fast Instead of Paying the Full Cost Every Time

class CircuitBreaker:
    def call(self, func):
        if self.state == 'open':
            if time.time() - self.opened_at >= self.reset_timeout:
                self.state = 'half-open'
            else:
                raise RuntimeError('Circuit open - failing fast')
        try:
            result = func()
            if self.state == 'half-open': self.state = 'closed'; self.failure_count = 0
            return result
        except Exception:
            self.failure_count += 1
            if self.failure_count >= self.failure_threshold:
                self.state = 'open'; self.opened_at = time.time()
            raise
    

Verified directly — the breaker saves real, measured time once it opens

Against a downstream call that always fails after a simulated 0.1s timeout: 10 calls with no circuit breaker took 1,003.5ms total — every single call paid the full timeout cost. The identical 10 calls through a breaker (threshold 3) took 301.1ms — only the first 3 calls paid the full 0.1s cost; the remaining 7 failed instantly once the circuit opened. 3.3× less total time wasted, for the identical, genuinely-failed outcome.

Retries With Backoff: Giving a Struggling Service Room to Recover

Verified directly — a naive retry is a burst; exponential backoff genuinely spreads the load

Five retry attempts against a service that always fails: a naive immediate retry (no delay between attempts) completed all 5 attempts within 0.01ms — effectively a single instantaneous burst hitting the struggling service five times in a row. The identical 5 attempts with exponential backoff (0.05 × 2ⁿ seconds between tries) spread across 751.16ms — roughly 117,000× more real time between the first and last attempt, for the same 5 total tries.

Why "just retry" can make an incident worse

A struggling service under real load doesn't need five more requests arriving in the same millisecond — that's Chapter 1's own overload finding, self-inflicted. Backoff doesn't reduce how many times a client tries; it changes when those tries land, giving whatever's struggling downstream genuine breathing room between attempts instead of compounding the problem that caused the failure in the first place.

Bulkheads: Isolating Resources So One Failure Doesn't Starve Everything

Verified directly — a shared resource pool lets a non-critical dependency block a critical one

A shared connection pool (capacity 5): a slow, non-critical recommendations service acquires all 5 connections and holds them. A critical checkout request, using the same pool, then tries to acquire a connection and correctly gets False — blocked, purely because an unrelated, lower-priority feature exhausted a resource checkout also needed.

Verified directly — isolated pools keep the critical path available

The identical scenario, but with recommendations and checkout each given their own pool (3 and 2 connections respectively): recommendations still exhausts its own pool completely (a 4th request correctly returns False) — but checkout's own separate pool is untouched, and its request correctly returns True. The critical path stayed available specifically because it was never sharing a resource with the failing one.

Graceful Degradation: A Usable Response, Not a Failed One

Verified directly — a resilient gateway returns a usable partial response when one dependency is down

Reusing Chapter 8's own aggregating gateway, with ReviewService genuinely failing: a resilient version, catching that specific failure and substituting a fallback ({'reviews': [], 'note': 'reviews temporarily unavailable'}), correctly returns the full order and stock data plus the fallback — a real, usable response. A brittle version, with no such handling, correctly fails the entire request — order and stock data included, even though both of those were computed successfully before the review call ever failed.

Where This Connects

This chapter's finding	What it connects to
A circuit breaker saving 3.3× total time on a genuinely failing call chain	Software Architecture Fundamentals Chapter 4's own cascading-call finding — this is the direct, concrete fix for the exact failure that chapter measured
A verified retry-storm vs. spread-out backoff, in real measured milliseconds	Chapter 7's own rate-limiting chapter — both are ways of controlling request density against a downstream system, one from the caller's side, one from the receiver's
A brittle gateway discarding two successful results because a third call failed	Technical Support's own `incident1`/`perfdiag1` — this exact pattern (one failed dependency taking down an otherwise-healthy response) is precisely the class of ticket those courses' own diagnostic chapters are built to trace back to its cause

Hands-On Exercises

Exercise 1

Using this chapter's own CircuitBreaker, lower failure_threshold to 1 and re-run the 10-call scenario. Verify the total time drops further than this chapter's own 301.1ms result, and explain the tradeoff a threshold of 1 introduces that a threshold of 3 avoids.

📄 View solution

Exercise 2

Using this chapter's own bulkhead scenario, give recommendations a pool of 4 instead of 3 (checkout stays at 2, for a combined total of 6 — one more than the original shared pool's own capacity of 5). Verify checkout still succeeds regardless of how large recommendations' own pool is.

📄 View solution

Exercise 3

Using this chapter's own four verified patterns, explain which ONE of them would have been the most direct fix for Software Architecture Fundamentals Chapter 4's own original cascading-call scenario (service A waiting 300ms because service C was slow), and why the other three, while genuinely useful in general, wouldn't have addressed that specific measured problem.

📄 View solution

Chapter 9 Quick Reference

Circuit breaker: stops calling a service that's genuinely failing — verified: 3.3× less total time wasted, 7 of 10 calls failing instantly instead of paying a full timeout
Retries with backoff: spaces out retries instead of bursting them — verified: 751ms spread vs. a naive retry's 0.01ms burst for the same 5 attempts
Bulkheads: isolate resource pools per dependency — verified: a shared pool let one feature block another; separate pools kept the critical path available
Graceful degradation: return a usable partial response instead of failing everything — verified: a resilient gateway kept two working results when a third dependency failed; a brittle one discarded all three
Next chapter: Capstone — designing a real system at scale, applying every chapter in this course together