Exercise 2: An A/A Test Sanity Check — Possible Solution ==================================================================== GIVEN ------------------------------ Group 1: n1=25, mean1=200, s1=30 Group 2: n2=28, mean2=215, s2=35 STEP 1: THE STANDARD ERROR OF THE DIFFERENCE ------------------------------ SE_diff = sqrt(s1^2/n1 + s2^2/n2) = sqrt(30^2/25 + 35^2/28) = sqrt(900/25 + 1225/28) = sqrt(36 + 43.75) = sqrt(79.75) ~= 8.930 STEP 2: THE T-STATISTIC ------------------------------ t = (mean2 - mean1) / SE_diff = (215 - 200) / 8.930 ~= 1.680 STEP 3: THE CONSERVATIVE DEGREES OF FREEDOM ------------------------------ df = min(n1, n2) - 1 = min(25, 28) - 1 = 24 STEP 4: DOES THE A/A TEST PASS ITS OWN SANITY CHECK? ------------------------------ Given critical value at two-tailed alpha=0.05, df=24: 2.064 |t| = 1.680, which is less than 2.064. Per this chapter's own decision rule, we FAIL TO REJECT H0 - there is no statistically significant difference between the two identically- treated groups. This A/A test PASSES its own sanity check: the traffic-splitting mechanism shows no detectable artificial bias, so a real A/B test can proceed with confidence in the underlying infrastructure, exactly matching this chapter's own Step 4 outcome. WHY THIS WORKS AS AN ANSWER ------------------------------ Each quantity is computed using this chapter's own Step 4 Welch's- t-test method, and the sanity-check interpretation is drawn directly from comparing |t| against the given critical value, correctly identifying a "fail to reject" result as the DESIRED outcome for an A/A test specifically (unlike a real A/B test, where failing to reject would instead mean no detected effect).