Load Balancing

Distributed Systems & Scalability

Chapter 2 · Load Balancing

Chapter 1 showed horizontal scaling only helps once work can genuinely be split across machines. Load balancing is the part that actually does the splitting — and how it splits requests turns out to matter as much as whether it splits them at all.

Round Robin vs. Least Connections, With Unequal Servers

class RoundRobinBalancer:
    def __init__(self, servers): self.servers = servers; self.index = 0
    def route(self):
        server = self.servers[self.index % len(self.servers)]
        self.index += 1
        server.enqueue(); return server

class LeastConnectionsBalancer:
    def __init__(self, servers): self.servers = servers
    def route(self):
        server = min(self.servers, key=lambda s: s.queue)
        server.enqueue(); return server
    

Three servers — two fast (capacity 5 requests/tick), one genuinely slow (capacity 2/tick) — under 12 new requests every tick for 20 ticks:

Verified directly — Round Robin lets the slow server's queue grow unboundedly

Under Round Robin, which sends exactly a third of all traffic to the slow server regardless of how backed up it gets, the slow server's queue reached 40 outstanding requests and never drained — it was still growing when the simulation ended. Under Least Connections, which always routes to whichever server currently has the fewest queued requests, the slow server's queue never exceeded 3. Both balancers received the identical total request volume — the only difference was which server each new request landed on.

Round Robin isn't broken — it's answering a different question

Round Robin guarantees each server gets an equal share of requests. Least Connections guarantees each server gets a roughly equal amount of outstanding work. Those are the same thing only when every server processes requests at the same speed — Chapter 1's own horizontal-scaling findings already showed that assumption doesn't always hold, even for identical hardware under different load.

Health Checks: Routing Around a Server That's Actually Down

Verified directly — a health-blind balancer kept failing requests after a server went down; a health-checking one didn't

Simulating server B going down partway through 30 requests: a plain Round Robin balancer with no health checking kept routing roughly a third of all remaining requests to the dead server, producing 7 failed requests out of 30. The identical scenario, routed through a balancer that filters to only currently-healthy servers before choosing one, produced 0 failed requests — every request was automatically redirected to A or C instead.

A lighter-weight cousin of Design Patterns' own State

This chapter modeled server health as a simple healthy boolean, checked with a plain if. Design Patterns Chapter 8 modeled a genuinely richer set of behaviors — an order's status — as full state objects, each owning its own transition rules. If a real health-check system needed more than "route or don't" (say, a "draining" state that finishes existing connections but accepts no new ones), the boolean would stop being enough, and reaching for that same State pattern would be the natural next step — not a different idea, just a heavier tool for a genuinely more complex version of the same problem.

Sticky Sessions: Solving Statelessness's Problem, Creating a New One

Software Architecture Fundamentals Chapter 8 verified a stateful design losing data on a server restart, and a stateless one surviving it. Sticky sessions are the load-balancer-level alternative: pin a client to the same server for their whole session, so that server can hold state in memory safely. What does pinning cost?

def sticky_route(session_id):
    h = int(hashlib.md5(session_id.encode()).hexdigest(), 16)
    return SERVERS[h % len(SERVERS)]  # pinned for the session's whole lifetime
    

Verified directly — sticky sessions produced a genuinely uneven split from nothing but hash luck

Hash-pinning 30 distinct client sessions across 3 identical servers, then sending 5 requests per client (150 total): the resulting split was A: 60, B: 65, C: 25 — a spread of 40 between the busiest and quietest server, despite every server being identical and every client sending exactly the same amount of traffic. Routing the identical 150 requests through Least Connections instead — no stickiness, no pinning — produced a perfectly even A: 50, B: 50, C: 50, a spread of 0.

The real tradeoff, stated precisely

Sticky sessions buy back the option of stateful, in-memory design that Software Architecture Fundamentals Chapter 8 verified breaking under a stateless assumption — but they do it by giving up the load balancer's own ability to route around uneven load, verified above producing a 40-request spread from just 30 clients. This is exactly the kind of tradeoff Chapter 8 itself flagged: neither option is free, and which one to accept depends on what the system actually needs more.

Where This Connects

This chapter's finding	What it connects to
An unbounded queue under Round Robin with unequal server speeds	Chapter 1's own horizontal-scaling findings — unequal processing capacity is exactly the condition that makes a "fair share of requests" different from "fair share of work"
Zero failures with health checking vs. 7 without	Chapter 9's own Fault Tolerance & Resilience Patterns — health checking is the first, simplest resilience pattern this course covers
Sticky sessions' verified 40-request spread from pure hash luck	Software Architecture Fundamentals Chapter 8's own statelessness finding — the two chapters verify opposite sides of the identical tradeoff

Hands-On Exercises

Exercise 1

Re-run this chapter's own Round Robin vs. Least Connections simulation with two slow servers (capacity 2/tick) and only one fast server (capacity 5/tick). Verify whether Least Connections still keeps queues bounded, and report the new queue depths for both balancers.

📄 View solution

Exercise 2

Using this chapter's own health-checking simulation, make two of the three servers go down at different points (B at request 10, C at request 20). Verify the health-checking balancer still produces zero failures, and report how many requests land on the one server left standing.

📄 View solution

Exercise 3

Using this chapter's own two verified findings (unequal-speed Round Robin, and sticky sessions), explain what specifically these two scenarios have in common: why does "give every option an equal share" produce a worse outcome than "route based on current state" in both cases?

📄 View solution

Chapter 2 Quick Reference

Round Robin: equal share of requests — verified: let a slow server's queue grow to 40 and keep climbing, when server speeds genuinely differ
Least Connections: routes to whichever server has the least outstanding work — verified: kept the same slow server's queue capped at 3
Health checks, verified: 7 failed requests without them, 0 with them, for the identical mid-run server failure
Sticky sessions, verified: a 40-request spread from pure hash luck across 3 identical servers, versus 0 spread for the same traffic under Least Connections — the direct tradeoff against Software Architecture Fundamentals Chapter 8's own statelessness finding
Next chapter: Caching Strategies — cache-aside, write-through, write-behind, and why invalidation is the genuinely hard part