Exercise 3: Expected False Positives Across 20 Null Tests — Possible Solution ==================================================================== GIVEN ------------------------------ 20 independent A/B tests, alpha = 0.05, none of the 20 features has any real effect (H0 is genuinely true for all 20). STEP 1: THE EXPECTED NUMBER OF FALSE POSITIVES ------------------------------ Per this chapter's own multiple-testing finding, each individual test has exactly a 5% (alpha) chance of producing a "statistically significant" result purely by chance, even though H0 is genuinely true. Across 20 independent such tests: Expected significant results = 20 x 0.05 = 1.0 So on average, about 1 out of the 20 tests would show a "significant" result purely by chance, with none of the underlying features actually doing anything. WHAT THIS IMPLIES FOR "3 OUT OF 20 WERE SIGNIFICANT" ------------------------------ A report of 3 significant results out of 20 tests is three times higher than the roughly 1 false positive expected by chance alone under this specific scenario (all truly null). This is a meaningful signal that at least some of those 3 results may reflect genuine effects rather than being purely coincidental - but it is NOT automatic proof that all 3 are real either, since getting 3 false positives out of 20 purely by chance, while less likely than getting 1, is still entirely possible with random variation. The practical implication, per this chapter's own multiple-testing caution, is that a team should never simply trust every individual "significant" result out of a large batch of tests at face value - the raw count of significant results needs to be interpreted against the number expected by chance alone (here, about 1), and ideally followed up with a proper multiple-testing correction or independent replication of the specific results that looked interesting, rather than reporting all 3 as confirmed wins without further scrutiny. WHY THIS WORKS AS AN ANSWER ------------------------------ The expected false-positive count is computed directly using this chapter's own multiple-testing formula (number of tests x alpha), and the interpretation of the reported 3-out-of-20 result is grounded in comparing it against that expected baseline rather than either dismissing all 3 results or accepting all 3 uncritically.