A/B Testing in Practice

Statistical Inference & Applied Statistics

Chapter 6 · A/B Testing in Practice

Chapter 5's own visible-but-not-significant response-time gap set up the exact question this chapter answers properly. Real A/B tests almost always compare a proportion — a conversion rate, a click-through rate — not a continuous mean. That needs the proportion-specific version of everything built so far.

The Two-Proportion Z-Test

Comparing two independent conversion rates uses two different standard error formulas, depending on the question:

For the hypothesis test (H₀: p₁ = p₂) — pooled SE

p̂ = (x₁+x₂)/(n₁+n₂) SE_pooled = √(p̂(1−p̂)(1/n₁ + 1/n₂)) z = (p₁−p₂)/SE_pooled

For the confidence interval on the difference — unpooled SE

SE_diff = √(p₁(1−p₁)/n₁ + p₂(1−p₂)/n₂) CI = (p₁−p₂) ± z* × SE_diff

The distinction matters: the hypothesis test assumes H₀ is true (the two groups share one real rate, so it's honest to pool them into a single best estimate for that shared rate). The confidence interval makes no such assumption — it uses each group's own separate observed rate, since it's estimating how different the two groups genuinely are, not testing whether they're the same.

A validity check worth doing before trusting the normal approximation

This entire z-test relies on the normal approximation to the binomial distribution (Probability & Statistics Fundamentals Chapter 6). The standard rule of thumb: both n·p and n·(1−p) should be at least 5 in each group, or the approximation can break down for rare events.

Worked Example: A Conversion-Rate A/B Test

Control (A): n₁=1000, 80 conversions (p₁=8%). Treatment (B): n₂=1000, 100 conversions (p₂=10%). Validity check: 1000×0.08=80, 1000×0.92=920, both comfortably above 5 for both groups — the normal approximation is safe to use.

Quantity	Value
p̂ (pooled)	180/2000 = 0.09
SE_pooled	≈ 0.01280
z	(0.10−0.08)/0.01280 ≈ 1.563
p-value (two-tailed)	≈ 0.118
Decision at α=0.05	0.118 > 0.05 → fail to reject H₀

The 95% confidence interval for the true difference, using the unpooled SE: 0.02 ± 1.96 × 0.01279 ≈ [−0.005, 0.045] — a range spanning from a small negative effect to a fairly large positive one, consistent with the test's own failure to reach significance. The interval crossing zero and the non-significant p-value are two views of the exact same conclusion.

Sample Size Planning — Was This Test Big Enough?

Statistical power (1 − β, where β is Chapter 4's own Type II error rate) is the probability a test correctly detects a real effect of a given size, if one truly exists. Before running a test, sample size can be planned to hit a target power (conventionally 80%):

Sample size formula (per group, two-proportion test)

n ≈ 2(z_α/2 + z_β)² × p̄(1−p̄) / (p₁−p₂)², with z_α/2 = 1.96 (95% confidence) and z_β ≈ 0.84 (80% power) as standard published constants

For the same 8%→10% effect actually being tested above (p̄ = 0.09):

The test was underpowered — this explains the non-significant result

n ≈ 2(1.96+0.84)² × 0.09(0.91) / (0.02)² ≈ 3,210 per group. The actual test only had n = 1,000 per group — less than a third of what proper planning would have called for. This is almost certainly why the test failed to reach significance, even if the treatment genuinely does help: the sample was simply too small to reliably detect a real 2-percentage-point effect at this base rate, not proof that no real effect exists.

Why Smaller Effects Need Dramatically More Data

The effect size sits squared in the denominator of the sample size formula — a small change in the effect being detected has an outsized impact on the data required.

Effect to detect	Required n per group (80% power, α=0.05)
8% → 10% (2 percentage points)	≈ 3,210
8% → 9% (1 percentage point)	≈ 12,195

Halving the effect size to detect very nearly quadrupled the required sample size — a direct consequence of that squared denominator, and a genuinely important planning reality: reliably detecting small improvements is expensive in data, often far more than intuition suggests.

A Real Trap: Peeking at Results Early

Stopping as soon as it "looks significant" inflates the false-positive rate

Checking a test's p-value repeatedly while it's still running, and stopping the moment it dips below 0.05, is a well-documented way to badly inflate the true false-positive rate above the intended α — related to, but distinct from, Chapter 4's own multiple-testing trap. A p-value naturally wanders up and down as data accumulates; checking it many times gives many chances for it to randomly dip below 0.05 at some point, even with zero real effect. The correct practice is to decide the sample size before the test starts (using the planning formula above) and only check the result once that predetermined size is reached.

A/B Testing in Code

import math

def standard_normal_cdf(z):
    return 0.5 * (1 + math.erf(z / math.sqrt(2)))

def two_proportion_z_test(x1, n1, x2, n2):
    p1, p2 = x1 / n1, x2 / n2
    p_pool = (x1 + x2) / (n1 + n2)
    se_pooled = math.sqrt(p_pool * (1 - p_pool) * (1/n1 + 1/n2))
    z = (p2 - p1) / se_pooled
    p_value = 2 * (1 - standard_normal_cdf(abs(z)))
    return z, p_value

z, p = two_proportion_z_test(x1=80, n1=1000, x2=100, n2=1000)
print(z, p)  # 1.563, 0.118 — matches the worked example

def required_sample_size(p1, p2, z_alpha2=1.96, z_beta=0.84):
    pbar = (p1 + p2) / 2
    return 2 * (z_alpha2 + z_beta)**2 * pbar * (1 - pbar) / (p2 - p1)**2

print(required_sample_size(0.08, 0.10))  # ~3210 per group

Hands-On Exercises

Exercise 1

Control: n₁=500, 60 conversions. Treatment: n₂=500, 85 conversions. First check the normal-approximation validity condition, then compute the pooled proportion, the z-statistic, and the p-value. State the decision at α=0.05.

📄 View solution

Exercise 2

Using this chapter's own sample size formula, compute the required sample size per group (80% power, α=0.05) to detect a change from a 5% baseline conversion rate to a 6% conversion rate (a 1-percentage-point effect).

📄 View solution

Exercise 3

A team plans a test for a fixed 4-week sample size, but decides to check the p-value every single day and stop the test as soon as it first drops below 0.05. Using this chapter's own peeking finding, explain why this practice produces a real false-positive rate higher than the intended 5%, even if the team genuinely stops testing the instant they see a "significant" result.

📄 View solution

Chapter 6 Quick Reference

Two-proportion z-test: pooled SE for the hypothesis test (H₀: p₁=p₂), unpooled SE for the confidence interval on the difference
Validity check: n·p ≥ 5 and n·(1−p) ≥ 5 in each group before trusting the normal approximation
Statistical power (1−β) is the probability of correctly detecting a real effect — plan sample size for it before running a test
Sample size formula: n ≈ 2(z_α/2+z_β)²p̄(1−p̄)/(p₁−p₂)² — effect size is squared in the denominator, so halving the effect roughly quadruples the required data
A non-significant result on an underpowered test doesn't prove there's no effect — it may just mean there wasn't enough data to detect one
Never peek and stop early — checking repeatedly and stopping at the first "significant" reading inflates the true false-positive rate well above α
Next chapter: Correlation vs. causation