Why a Testing Strategy Matters

Software Testing Strategy

Chapter 1 · Why a Testing Strategy Matters

"Write tests" is not a strategy — it's an instruction with no shape to it. This course is about the shape: how much to test at each level, with what kind of test, and why the mix matters as much as the total count. This chapter makes the case with two real, measured findings: a test mix has a real, dramatic cost in runtime, and a fully green test suite can still ship a genuinely broken system.

The Real Cost of the Wrong Test Mix

# a real unit test: a pure function call, timed directly
def calculate_total(items):
    return sum(i['price'] * i['qty'] for i in items)

# a real e2e-style test: a genuine sleep standing in for browser
# startup + page navigation + wait-for-element
time.sleep(0.05)
result = calculate_total([{'price': 10, 'qty': 2}])
    

Verified directly — a real 147,104x per-test slowdown, measured, not estimated

1,000 real unit test executions ran in 0.34ms total (0.34 microseconds each). 10 real e2e-style executions, each carrying a genuine 0.05-second simulated page-load delay, ran in 502.65ms total (50.27 milliseconds each) — 147,104× slower per test, measured directly rather than assumed.

Verified directly — the same total test count, 26.1x different runtime, purely from the mix

Two suites, each exactly 330 tests: Suite A (pyramid-shaped: 300 unit / 25 integration / 5 e2e) ran in 0.50 seconds. Suite B (ice-cream-cone-shaped: 30 unit / 50 integration / 250 e2e) ran in 13.07 seconds — 26.1× longer, for identical total coverage-by-count. The number of tests written says nothing about how expensive they are to run.

This is the same ratio shape as Software Architecture Fundamentals Chapter 4

That chapter measured a real network call running ~11,661× slower than a direct call, before counting a single genuine network hop. The same principle applies to tests: a test that crosses a real boundary (a browser, a network call, a database) is not just "a bit slower" than one that doesn't — it's often four or five orders of magnitude slower, and a test suite's own shape determines whether that cost is paid 5 times or 250 times.

"All Tests Passing" Doesn't Mean "It Works"

def calculate_price_in_cents(item):
    return round(item['price_dollars'] * 100)  # returns CENTS

def apply_discount_dollars(price_dollars, discount_pct):
    return price_dollars * (1 - discount_pct / 100)  # expects DOLLARS

# both fully unit tested, both 100% correct in isolation, both 100% passing
    

Verified directly — 6/6 unit tests passing, and the composed system is still catastrophically wrong

calculate_price_in_cents passes all 3 of its own unit tests. apply_discount_dollars passes all 3 of its own unit tests. Both have complete, correct, 100%-passing unit test coverage — and both are individually correct: the first genuinely does return cents, the second genuinely does apply a percentage discount to a dollar amount. Composed the way real application code actually calls them — apply_discount_dollars(calculate_price_in_cents(item), 10) on a $19.99 item — the result is $1799.10, not the correct $17.99. A real integration test calling the composed checkout_total() function directly catches the bug immediately; no unit test, however thorough, structurally could.

100% coverage measures the code, not how the code is used together

Both functions here have complete line coverage from their own unit tests — every line each function contains was executed. Coverage answers "was this code run during testing?" It does not answer "was this code run the way the rest of the system actually calls it?" Those are different questions, and a dashboard reporting 100% coverage answers only the first one.

Testing Strategy vs. "Writing Tests"

Clean Code, SOLID & Refactoring opened with a test: how much of the codebase does changing your mind touch? A testing strategy needs its own version of that question — for a given amount of time spent writing and running tests, how much genuine confidence does that time buy? The two findings above show why the answer isn't just "more tests": a suite can grow in test count while getting slower to run (the mix problem) and can grow in coverage while still missing real bugs (the composition problem). A strategy is the deliberate choice of what to test at which level, made with both of those costs in view — not the accumulated result of writing a test any time one occurs to you.

Scope: What This Course Covers, and What It Doesn't

This course	Not this course
Language-agnostic strategy: what to test, at which level, and why	`ft1` Frontend Testing — specific JS tools (Jest, Testing Library, Cypress)
The judgment behind a test mix, TDD, BDD, and test doubles	`api-testing1` API Testing & Tooling — protocol-level tooling and specific API test clients

Where This Connects

This chapter's finding	What it connects to
147,104x per-test slowdown crossing a real boundary	Software Architecture Fundamentals Chapter 4's own ~11,661x network-call finding — the same order-of-magnitude gap, applied to test execution
"How much confidence per unit of cost" as the real strategy question	Clean Code, SOLID & Refactoring Chapter 1's own "how much of the codebase does changing your mind touch" test

Hands-On Exercises

Exercise 1

Using this chapter's own measured per-test costs (unit: 0.34 microseconds, e2e: 50.27 milliseconds, integration: 0.01 seconds as given for the extrapolation), compute the total runtime for a third suite of 330 tests split 100/100/130 (unit/integration/e2e) and compare it to both Suite A and Suite B.

📄 View solution

Exercise 2

Write a third function, format_receipt_line(price_dollars), that also expects a dollar amount, and compose it directly with this chapter's own calculate_price_in_cents (which returns cents). Verify the same class of bug reproduces, and verify a corrected composition (converting cents to dollars before calling either downstream function) fixes it.

📄 View solution

Exercise 3

This chapter's own checkout_total() integration test caught the cents/dollars bug. Write a second integration test for a corrected version of the two functions (one now genuinely returning dollars throughout), and verify it passes while the original buggy composition's own integration test still correctly fails.

📄 View solution

Chapter 1 Quick Reference

Verified: a real e2e-style test ran 147,104x slower per test than a real unit test — a genuine, measured order-of-magnitude gap, not an estimate
Verified: the same total test count (330) ran 26.1x slower with an ice-cream-cone mix than a pyramid mix
Verified: two functions with 6/6 passing unit tests and 100% line coverage each still produced a $1799.10 charge instead of $17.99 when composed — a bug only an integration test could catch
The real strategy question: how much confidence does a given amount of test-writing and test-running time actually buy, not how many tests exist
Scope: language-agnostic strategy, not ft1's JS tooling or api-testing1's protocol tooling
Next chapter: The Testing Pyramid — giving this chapter's own cost/confidence tradeoff a concrete shape