Exercise 3: Why Daily Peeking Inflates the False-Positive Rate — Possible Solution ==================================================================================== WHY A SINGLE PLANNED CHECK KEEPS ALPHA AT 5% ------------------------------ Per this chapter's own decision rule, alpha=0.05 means there's a 5% chance of a false positive on any SINGLE hypothesis test performed under a truly null effect. If the team collects exactly 4 weeks of data as planned and checks the p-value exactly once at the end, that 5% figure is the genuine, honest false-positive rate for the whole experiment. WHAT CHANGES WHEN THE TEST IS CHECKED EVERY DAY ------------------------------ Checking the p-value every single day over a 4-week test means running roughly 28 separate significance checks on the SAME accumulating dataset, not just one. Per this chapter's own peeking finding, a p-value naturally fluctuates up and down as more data comes in throughout a test, even when there is truly no real effect at all. Stopping the very first time that fluctuating p-value happens to dip below 0.05 is functionally very similar to Chapter 4's own multiple- testing trap - with 28 separate opportunities for the p-value to randomly cross the 0.05 threshold at some point, the overall chance that at least one of those 28 checks shows "significance" purely by chance is considerably higher than 5%, even though each individual day's snapshot might look like an ordinary 0.05-alpha check in isolation. WHY THIS HAPPENS EVEN IF THE TEAM STOPS THE INSTANT THEY SEE SIGNIFICANCE ------------------------------ The inflation comes from the NUMBER OF CHANCES the p-value gets to dip below 0.05 at some point during the accumulating data, not from anything about how quickly the team reacts once it does. Stopping immediately doesn't undo the fact that the p-value was checked (and could have triggered a stop) many separate times before that - the real false-positive rate is a property of the whole day-by-day checking PROCESS, not of the single day it happened to cross the threshold. WHY THIS WORKS AS AN ANSWER ------------------------------ The explanation is grounded directly in this chapter's own peeking finding, connects it explicitly to the closely related multiple- testing mechanism from Chapter 4, and specifically addresses why stopping immediately upon seeing significance does not prevent the inflation, rather than only restating that peeking is risky in general terms.