Why Systems Need to Scale

Distributed Systems & Scalability

Chapter 1 · Why Systems Need to Scale

Software Architecture Fundamentals answered "how should this system's own code be organized?" This course answers a genuinely different question: once that system is correctly organized, what happens when real load hits it? A correctly-layered, correctly-bounded architecture (Software Architecture Fundamentals' own OrderService/OrderRepository/PricingEngine) doesn't automatically survive scale — this chapter measures two completely different reasons why.

Reason 1: An Algorithmic Bottleneck, Hiding in Correct-Looking Code

A duplicate-order-ID check is a natural thing to add to OrderRepository. The obvious implementation scans every existing order — and that "obvious" choice is the actual bottleneck.

def order_id_exists_linear(order_ids_list, order_id):
    return order_id in order_ids_list  # O(n) — scans the list

def order_id_exists_set(order_ids_set, order_id):
    return order_id in order_ids_set  # O(1) average case — hash lookup
    

Verified directly — the linear check's cost grows with data size; the hash check's doesn't

Checking for the last-inserted order ID (the worst case for a linear scan) against a growing order list: at 1,000 orders, the linear check took 7.78 microseconds versus the set check's 0.075 microseconds — 104× slower. At 20,000 orders, the linear check grew to 141.88 microseconds while the set check stayed essentially flat at 0.080 microseconds — now 1,785× slower. The linear check's own cost grew roughly in proportion to the order count; the set check's didn't move at all.

This is not a scaling problem — and no amount of scaling fixes it

Buying a faster server (vertical scaling) makes every microsecond figure above smaller, but the growth curve stays exactly the same shape — the linear check will still eventually outpace any fixed amount of extra speed as the order count keeps growing. Running the check on more machines (horizontal scaling) doesn't help either, unless the order list itself gets split up somehow — which raises its own hard question (Chapter 4's own database-sharding chapter). The actual fix here is Chapter 1's own lesson from Software Architecture Fundamentals, one level deeper: sometimes what "breaks first" isn't the architecture's own shape, it's one specific algorithm hiding inside a correctly-organized component.

Reason 2: A Single Process Can Only Use So Much Hardware

Even a perfectly-written PricingEngine runs as ordinary Python code in one process. Genuinely parallelizing it — running several calculations at once, on several CPU cores — needs more than just "a bigger server."

def cpu_heavy_pricing_calc(base_price):  # genuine CPU-bound work, not a sleep() simulation
    total = base_price
    for i in range(200000):
        total = (total * 1.0000001) % 100000
    return round(total, 2)
    

Verified directly — parallelizing a small workload made it genuinely slower

Running 8 pricing calculations sequentially took 0.069s. Running the same 8 across 4 processes (Python's own multiprocessing.Pool) took 0.207s — roughly 3× slower, not faster. Spinning up separate OS processes has real overhead, and for a workload this small, that overhead cost more than the parallelism saved.

Verified directly — a genuinely large workload does benefit, but sub-linearly

Running 160 calculations sequentially took 1.327s. The identical 160 calculations across 2 processes took 0.816s (1.63× faster); across 4 processes, 0.532s (2.49×); across 8 processes, 0.472s (2.81×). Doubling the workers from 2 to 4 improved things — but nowhere near doubled the speedup, and going from 4 to 8 barely moved the number at all. Every result matched the sequential version's own output exactly, confirming the parallel version computed the identical correct answers, just faster.

Vertical vs. horizontal, stated precisely

Vertical scaling — a faster CPU, more RAM — would have sped up every number in both findings above uniformly, small workload included, with zero code changes and zero risk of making anything slower. Horizontal scaling — more processes, more machines — only helped once the workload was large enough to be worth splitting up, and even then delivered real but diminishing returns, never the "8 workers = 8× faster" result a naive mental model would predict.

Where This Connects

This chapter's finding	What it connects to
A 1,785× algorithmic slowdown, invisible until measured at real scale	Chapter 4's Database Scaling — sharding a growing dataset is the production-scale version of fixing exactly this kind of bottleneck
Small-workload parallelism verified genuinely slower, not just "less beneficial"	Chapter 2's Load Balancing — routing overhead has the same shape of cost, worth measuring before assuming it's free
Sub-linear speedup even for a large, genuinely parallelizable workload	Technical Support's own `perfdiag1` — this chapter's own honest ceiling is exactly the kind of number that course's diagnostic chapters teach how to notice in a real production dashboard

Hands-On Exercises

Exercise 1

Repeat this chapter's own linear-vs-set benchmark, but check for an order ID that's first in the list rather than last. Verify the linear check's timing changes dramatically while the set check's doesn't, and explain why using this chapter's own O(n)/O(1) framing.

📄 View solution

Exercise 2

Re-run this chapter's own small-workload multiprocessing benchmark (8 calculations) with only 2 processes instead of 4. Verify whether the "parallelizing makes it slower" finding still holds, and report the actual speedup ratio.

📄 View solution

Exercise 3

Using this chapter's own two verified findings, explain why "just add more servers" is not a universal fix for a slow system — name the specific condition each finding shows has to be true before horizontal scaling actually helps.

📄 View solution

Chapter 1 Quick Reference

Vertical scaling: a more powerful single machine — helps uniformly, but has a ceiling and never fixes an algorithmic growth-rate problem
Horizontal scaling: more machines/processes working in parallel — verified: genuinely slower for a small workload (overhead dominates), genuinely faster but sub-linear for a large one (1.63×–2.81× across 2–8 processes)
Verified: a linear duplicate-check grew from 104× to 1,785× slower than an O(1) equivalent as data size grew from 1,000 to 20,000 — a bottleneck no amount of scaling alone fixes
Next chapter: Load Balancing — how requests actually get distributed once there's more than one server to send them to