Catastrophic Cancellation & Loss of Significance

Numerical Methods & Floating-Point Computation

Chapter 4 · Catastrophic Cancellation & Loss of Significance

Chapter 3 established machine epsilon — the smallest gap between representable values near 1.0. This chapter shows the single most common way that gap turns into a real, visible bug: subtracting two floating-point numbers that are very close to each other. The operation itself is completely correct by IEEE 754's own rules — it's what subtraction does to your remaining precision that causes the damage.

The Core Mechanism

Every floating-point number carries roughly 15-17 significant decimal digits of precision (Chapter 2's 52-bit mantissa). When two numbers that agree in their first several digits are subtracted, those matching leading digits cancel out exactly — but the trailing digits that made each number only approximately correct in the first place don't cancel; they're what's left over. The result can end up built almost entirely out of rounding error, even though the subtraction itself was performed with full precision.

Verified directly — precision collapsing as two numbers converge
Computing (1 + x) − 1, which is mathematically always exactly x, for shrinking values of x: x=10⁻⁸ gives a relative error of only 6.08 × 10⁻⁹ (still fine) — but x=10⁻¹⁵ gives a relative error of 0.11 (11% wrong), and at x=10⁻¹⁶ the result is exactly 0.0, a relative error of 1.0 — x has vanished completely. Nothing "went wrong" with the subtraction; 1+x simply rounded to exactly 1.0 before the subtraction ever happened, because x was smaller than machine epsilon relative to 1.0.

The Classic Case: An Unstable Quadratic Formula

The quadratic formula, x = (−b ± √(b² − 4ac)) / (2a), is taught as a single, universal recipe — but implemented naively in floating point, it has a real, well-known failure mode whenever b is much larger than a and c: one of the two ± branches subtracts two nearly-equal numbers.

# a=1, b=-100000, c=1 --> roots near 99999.99999 and 0.00001 disc = b*b - 4*a*c sq = math.sqrt(disc) root_big = (-b + sq) / (2*a) # fine: -b and sq have the same sign here root_small = (-b - sq) / (2*a) # danger: -b and -sq nearly cancel
Verified directly — the naive formula's small root is badly wrong
Comparing against a high-precision (50-digit) reference computation: the naive formula's large root (99999.99999) is essentially perfect — relative error ≈ 3.4 × 10⁻¹⁷, right at the limit of double precision. But its small root (≈ 1.0000003385357559 × 10⁻⁵) has a relative error of ≈ 3.4 × 10⁻⁷ — ten orders of magnitude worse than the large root, computed from the exact same inputs with the exact same formula. The only difference is which ± branch happened to subtract two nearly-equal numbers.

The standard fix — sometimes called the "Citardauq" formula (read the standard quadratic formula's name backward) — restructures the computation so the two large, same-signed quantities are always added, never subtracted, and the small root is obtained by division instead:

sign_b = 1 if b >= 0 else -1 q = -0.5 * (b + sign_b * sq) # always an addition of same-signed terms root1 = q / a root2 = c / q # the small root, via division -- no cancellation
Verified directly — the stable formula fixes exactly the broken root
The stable formula's large root matches the naive formula's (relative error ≈ 3.4 × 10⁻¹⁷ — no change, since that branch was never the problem). Its small root, computed as c / q instead of by subtraction, has a relative error of just ≈ 1.1 × 10⁻¹⁶ — essentially at the limit of double precision, roughly nine orders of magnitude better than the naive formula's 3.4 × 10⁻⁷ for the identical mathematical root. Same inputs, same underlying mathematics, dramatically different reliability — purely a function of which arithmetic operations were used to get there.

Resolved: Calculus & Optimization's Own Unexplained Finding

Calculus & Optimization Chapter 1 verified that numerical differentiation, (f(x+h) − f(x)) / h, gets more accurate as h shrinks — until it suddenly gets catastrophically worse. That course stated the observation honestly but left the mechanism for this course to explain. It is exactly the cancellation pattern above.

Verified directly — watching the numerator collapse, step by step
For f(x)=x² at x=3 (so f(x)=9.0): as h shrinks, f(x+h) gets closer and closer to 9.0, so the subtraction f(x+h) − f(x) loses more and more relative precision — its own relative error grows from 7.7 × 10⁻⁹ at h=10⁻⁸ to 6.8 × 10⁻⁴ at h=10⁻¹³, and by h=10⁻¹⁶, f(x+h) rounds to exactly 9.0 — identical to f(x) — so the subtraction returns exactly 0.0, a complete, 100% loss of the numerator, even though the true difference was a tiny but genuinely nonzero 6 × 10⁻¹⁶.
Why dividing by a tiny h afterward makes it look even worse
Dividing by h doesn't add any new error — but it doesn't fix the numerator's damage either. It simply reveals it: the numerator's relative error is the final derivative approximation's relative error, since dividing by a number doesn't change relative error. A numerator that's already 100% wrong produces a derivative approximation that's 100% wrong — which is exactly why Calculus & Optimization's own experiment saw the approximation collapse to 0 at h=10⁻¹⁶, for a true derivative of 6. There was never a "division by a small number" problem on its own — the real damage happened one step earlier, in the subtraction.

Where This Connects

This chapter's findingWhat it resolves or sets up
Cancellation destroys precision when operands convergeFully resolves Calculus & Optimization Chapter 1's own unexplained numerical-differentiation breakdown
Same formula, wildly different reliability depending on arithmetic structureSets up Chapter 5's general treatment of algorithm stability — this chapter's quadratic-formula fix is the first concrete example of it
The stable "Citardauq" reformulation trades subtraction for addition + divisionA specific instance of the general reformulation strategy Chapter 9 applies again to numerical differentiation and integration

Hands-On Exercises

Exercise 1

Using this chapter's own (1+x)-1 experiment, explain precisely why the result is exactly 0.0 at x=10⁻¹⁶ rather than some small nonzero (if inaccurate) number. Your answer should reference machine epsilon from Chapter 3.

📄 View solution
Exercise 2

For the quadratic a=1, b=-100000, c=1, this chapter verified the naive formula's small root has a relative error roughly ten orders of magnitude worse than its large root. Explain specifically which arithmetic step causes the difference, and why the large root's computation never runs into the same problem.

📄 View solution
Exercise 3

Using this chapter's own step-by-step resolution of the numerical-differentiation breakdown, explain in your own words why "just use a smaller h" and "just use more decimal digits of precision" are not really two different potential fixes for the same problem, but are actually pulling in opposite directions once h gets small enough.

📄 View solution

Chapter 4 Quick Reference

  • Catastrophic cancellation: subtracting two nearly-equal floating-point numbers destroys relative precision, even though the subtraction itself is performed exactly per IEEE 754's own rules
  • Verified: (1+x)-1 for true value x goes from a negligible 6×10⁻⁹ relative error at x=10⁻⁸ to a complete, 100% loss (result=0.0) at x=10⁻¹⁶
  • The naive quadratic formula's small root, verified: relative error ≈3.4×10⁻⁷; the algebraically-equivalent stable ("Citardauq") reformulation: relative error ≈1.1×10⁻¹⁶ — nine orders of magnitude better, for the same math
  • The fix pattern: restructure the arithmetic so nearly-equal quantities are never directly subtracted — trade subtraction for addition-plus-division wherever the mathematics allows it
  • Resolved directly: Calculus & Optimization Chapter 1's numerical-derivative breakdown was cancellation in the numerator f(x+h)-f(x), not a problem with dividing by a small h
  • Next chapter: Numerical stability — generalizing this chapter's one worked fix into a broader way of recognizing unstable algorithms