Capstone — Designing and Analyzing a Real Experiment

Before designing anything, the team pulls 400 historical sessions: 32 conversions, p = 8%. This is a sample of past traffic, not a guaranteed future rate — it carries genuine sampling error, and past traffic mix may not perfectly match what the actual test window will see. This number is a starting estimate, not a fact to design blindly around.

Treating 8% as a proportion estimate: SE = √(0.08×0.92/400) ≈ 0.0136. The 95% confidence interval: 0.08 ± 1.96×0.0136 ≈ [5.3%, 10.7%]. The true baseline conversion rate could honestly be anywhere in that fairly wide range — a useful, humbling number to have before committing to a test design built on the single point estimate of 8%.

H₀: p_control = p_treatment (the button color makes no real difference) against H₁: p_control ≠ p_treatment, at the standard α = 0.05. Nothing about the experiment's own numbers is examined yet — the rules are fixed first, deliberately, to avoid the peeking trap Chapter 6 warned about.

Before trusting the traffic-splitting mechanism itself, the team runs an A/A test — two groups, both shown the old button, compared on average session duration to confirm the split itself introduces no artificial bias. Group 1: n=30, mean 145s, s=40s. Group 2: n=32, mean 138s, s=38s. Welch's t-test: SE_diff ≈ 9.92, t ≈ 0.705, df = 29, critical value ≈ 2.045. |0.705| < 2.045 → fail to reject, exactly the desired outcome — the split mechanism itself shows no detectable bias, so the actual A/B test can proceed with confidence in the infrastructure.

Using Step 1's own 8% baseline and wanting to detect a lift to 10% (Chapter 6's own worked effect size), proper sample-size planning calls for ≈3,210 per group — not the smaller, underpowered pilot Chapter 6's own example ran with. This time, the team runs the test to that full planned size: control n=3,210, 257 conversions (≈8.01%); treatment n=3,210, 321 conversions (10.00%).

Digging into the treatment group's data, engaged sessions (more product pages viewed) also tend to run longer — r ≈ 0.998 between pages viewed and session duration across a sample. It's tempting to conclude "showing users more product pages makes them stay longer," but per Chapter 7's own caution, this observational correlation could just as easily be reverse causation: genuinely interested users naturally view more pages and stay longer, with neither driving the other. Only Step 5's own randomized comparison — not this correlation — can support a causal claim.

Fitting a least-squares line to the same pages-viewed/duration data: duration ≈ 40.77 × pages + (−24.62), with R² ≈ 0.996 — a striking fit. Predicting duration for a session viewing 7 pages: ≈ 260.8s. Since the observed data only ranged up to 6 pages viewed, this prediction is already a mild extrapolation, per Chapter 8's own caution — worth flagging honestly rather than reporting with false confidence.

Step 5's result is a frequentist one: H₀ rejected at α=0.05, a binary decision. A Bayesian analyst, starting from a skeptical prior (most small UI tweaks historically move conversion very little), would update that prior using this experiment's own strong evidence (p≈0.0053, a properly powered result) and land on a substantially higher posterior belief the button change genuinely helps — a graded probability statement rather than a reject/fail-to-reject binary. Per Chapter 9's own honest scope boundary, computing that exact posterior number needs machinery (a specified prior distribution, conjugate updating) beyond this course — but the qualitative direction is clear either way: strong, properly powered evidence moves a skeptical starting belief substantially, regardless of which framework reports the final number.