End-to-End & System Testing

Software Testing Strategy

Chapter 6 · End-to-End & System Testing

The top of the pyramid tests the whole system the way a real user actually experiences it — every layer wired together, nothing doubled. That's exactly what makes it valuable, and exactly what makes it fragile. This chapter verifies both: a real, measured flakiness problem, a real bug only a full flow catches, and a real blind spot even a full flow can't touch.

Flakiness: A Measured, Not Anecdotal, Problem

def simulate_page_load():
    return random.uniform(10, 100)  # genuinely variable completion time, in ms

def naive_flaky_test():
    actual_load_time = simulate_page_load()
    fixed_wait = 30  # a guess at "probably long enough"
    return actual_load_time <= fixed_wait
    

Verified directly — a fixed 30ms wait produced a 23.5% pass rate over 200 identical runs

The exact same test, same code, same assertion, run 200 times against a genuinely variable 10-100ms load time: 47 passed, 153 failed — a 23.5% pass rate. Nothing about the test or the code under test changed between runs; only the random timing did. This is what "flaky" means precisely: a test whose result depends on something the test itself doesn't control.

Verified directly — polling for the real condition eliminated the flakiness entirely

Replacing the fixed wait with a loop that polls every 5ms up to a generous 200ms timeout, checking the actual condition each time rather than guessing how long it takes: 200 passed, 0 failed — a 100% pass rate, over the identical 200 simulated load times.

A flaky test is worse than no test

A test with a 23.5% pass rate doesn't communicate "the system is 76.5% broken" — it communicates nothing at all about the system, since it fails just as often when everything works correctly as when something is genuinely wrong. Teams that tolerate flaky tests tend to start ignoring failures generally, which means a real regression can hide inside the noise.

What Only a Full Flow Catches

def test_full_checkout_flow_e2e():
    token = session.login("u1")
    time.sleep(0.05)                        # realistic time browsing
    items = browse_catalog(session, token)  # still valid here
    time.sleep(0.15)                        # realistic time filling out shipping
    return checkout(session, token, items)  # session has now expired
    

Verified directly — every isolated per-step test passed; only the full flow caught the bug

test_login_isolated, test_browse_isolated, and test_checkout_isolated — each calling its own step immediately, with no elapsed time — all passed. The same three operations run as one continuous flow, with realistic delays between steps (matching how long a real user actually spends browsing and filling out a form): failed — "session expired during checkout."

This is a structural gap, not a testing gap

No unit test or integration test for checkout() alone could have caught this, however well written — the bug only exists in the passage of real time across steps, which an isolated test of any single step cannot represent by definition. A full end-to-end flow is the only test shape that naturally includes that elapsed time.

What Even a Full Flow Can't Catch

# reusing Distributed Systems & Scalability Chapter 9's own circuit breaker shape
def test_e2e_happy_path(breaker):
    result = breaker.call(healthy_dependency)  # every dependency healthy, throughout
    assert result == "OK"
    

Verified directly — a passing e2e happy-path test left the circuit breaker's own OPEN state completely untouched

test_e2e_happy_path passes. breaker.times_opened after the test: 0. The e2e test never once caused a failure, so it structurally cannot exercise what happens when one occurs — not because the test was written badly, but because a happy-path flow has no failure in it to trigger the breaker.

Verified directly — only a test that deliberately simulates repeated failure proves the resilience pattern itself

A dedicated test calling breaker.call(failing_dependency) three times in a row: breaker.state becomes "OPEN", times_opened becomes 1, and a subsequent call — even with a healthy dependency — is correctly rejected fast rather than attempted: "circuit open - failing fast."

E2e and resilience-pattern tests answer different questions

An e2e test answers "does the real user flow work when everything is healthy?" A resilience-pattern test answers "does the system behave correctly when something isn't?" Distributed Systems & Scalability's own Chapter 9 verified circuit breakers, backoff, bulkheads, and graceful degradation directly — none of that coverage comes for free from even a comprehensive e2e suite, because a healthy-path flow never has a reason to trigger any of it.

Where This Connects

This chapter's finding	What it connects to
A bug only visible across real elapsed time between steps	Chapter 2's own fault-localization tradeoff — a cost e2e coverage pays for, in exchange for catching this exact class of bug
A circuit breaker's OPEN state left completely unexercised by a happy-path e2e test	Distributed Systems & Scalability Chapter 9's own resilience-pattern verification — genuinely separate coverage, not a subset of e2e

Hands-On Exercises

Exercise 1

Using this chapter's own naive_flaky_test, increase the fixed wait from 30ms to 80ms (still less than the maximum possible 100ms load time) and re-run the same 200-iteration measurement. Report the new pass rate and explain why it's higher but still not 100%.

📄 View solution

Exercise 2

Modify this chapter's own full-flow e2e test so the total delay between login and checkout is 0.10 seconds instead of 0.20 seconds (still under the 0.15-second session timeout). Verify the flow now completes successfully, and explain why this doesn't mean the underlying session-expiry risk is gone.

📄 View solution

Exercise 3

Write a second resilience-pattern test for this chapter's own CircuitBreaker: after it trips to OPEN, simulate enough time passing for it to attempt a "half-open" retry (you'll need to add minimal half-open logic to CircuitBreaker yourself), and verify a subsequent successful call resets it back to CLOSED.

📄 View solution

Chapter 6 Quick Reference

Verified: a fixed-wait e2e test scored a 23.5% pass rate across 200 identical runs; polling for the real condition scored 100%
Verified: a session-expiry bug passed every isolated per-step test but failed the one test running the full flow with realistic elapsed time
Verified: a passing e2e happy-path test left a circuit breaker's OPEN state completely unexercised — 0 trips recorded
The flakiness fix: poll for the real condition instead of guessing a fixed wait time
What only e2e catches: bugs that exist in the passage of real time or state across multiple steps
What even e2e can't catch: resilience-pattern behavior — that needs tests that deliberately simulate failure, not just a healthy happy path
Next chapter: Test-Driven Development — writing the test before the code, and what that changes