Exercise 2: Why the Two-Pass Formula's Subtraction Isn't Dangerous — Possible Solution ==================================================================== WHAT MAKES A SUBTRACTION DANGEROUS, PRECISELY ------------------------------ Chapter 4 established that catastrophic cancellation happens specifically when subtracting two numbers that are close to each other IN VALUE - not just "any subtraction whatsoever." The danger comes from the operands sharing many leading digits that cancel out, leaving a result built mostly from each operand's own rounding error. A subtraction between two very different-sized numbers, or between numbers that are already small, doesn't have this problem, because there's little or nothing large to cancel. WHY x - mean IS A SAFE SUBTRACTION ------------------------------ In the two-pass formula, mean is the average of the ENTIRE dataset, and x is one specific data point drawn from that same dataset. By definition, x and mean are already close to each other in absolute terms relative to the data's own offset - but crucially, this subtraction happens BEFORE any large sums have been built up. The result of x - mean is a small number representing how far this one point deviates from the center of the data - exactly the same size as the data's own spread (in this chapter's own example, values around 0 to 1), not the enormous ~20,000,000 magnitude of the raw data itself. WHY THIS IS THE OPPOSITE SITUATION FROM THE NAIVE FORMULA ------------------------------ The naive formula's dangerous subtraction, E[X^2] - (E[X])^2, happens AFTER building two enormous intermediate sums (each around 4*10^14 in this chapter's own example), and subtracts those two huge, nearly-equal numbers to try to recover a tiny result. The two-pass formula never builds any large intermediate quantity in the first place - it strips away the large offset immediately, at the very first step, so every subsequent operation (squaring, summing) works only with already-small numbers. There is no large cancellation left to happen, because the large-magnitude offset was removed before it had any chance to hide error inside a bigger sum. WHY THIS WORKS AS AN ANSWER ------------------------------ The explanation correctly identifies that "involves a subtraction" is not the same as "is numerically dangerous" - what matters is whether the subtracted operands are large numbers that mostly cancel out. It contrasts the size of the two formulas' intermediate values directly (large sums vs. small deviations) rather than treating subtraction itself as the hazard.