Rounding Error & Machine Epsilon
Numerical Methods & Floating-Point Computation
Chapter 3 · Rounding Error & Machine Epsilon
Chapter 2 explained why a value like 0.1 gets rounded when it's stored. This chapter puts an exact number on how much that rounding can possibly be — the single most important quantity in this whole course for reasoning precisely about floating-point trustworthiness, rather than just gesturing at "some small error."
Absolute Error vs. Relative Error
Absolute error is simply |approximation − true value| — the raw size of the gap. Relative error divides that gap by the true value's own size: |approximation − true value| / |true value|. The same absolute error can mean something completely different depending on the scale of the number involved.
1,000,000.0, approximation 1,000,000.0001 — absolute error ≈ 0.0001, but relative error ≈ 1.0 × 10⁻¹⁰ (utterly negligible). Case 2: true value 0.0001, approximation 0.0002 — absolute error is the exact same 0.0001, but relative error is 1.0 — the approximation is 100% wrong, literally double the true value. Absolute error alone told you nothing useful; the two cases look identical by that measure and are utterly different in reality.
Machine Epsilon, Derived — Not Quoted
Machine epsilon is the smallest positive number ε such that 1.0 + ε is a different, representable floating-point value from 1.0 itself. Rather than simply stating its value, it can be found directly with a genuinely simple experiment: start with ε = 1.0 and keep halving it until adding it to 1.0 stops making a difference.
1.0 + ε stops changing. The last value of ε for which 1.0 + ε != 1.0 was still true is 2.220446049250313 × 10⁻¹⁶ — which is exactly Python's own built-in sys.float_info.epsilon, and exactly 2⁻⁵². This isn't a coincidence: it's the direct, mechanical consequence of Chapter 2's own 52-bit mantissa — 2⁻⁵² is precisely the size of the smallest possible adjustment representable at a magnitude around 1.0, and the halving experiment rediscovers that fact from first principles rather than being told it.
Machine epsilon is not "the smallest number a computer can represent" — Chapter 2's subnormal numbers go vastly smaller than this. It specifically measures precision relative to a value near 1.0 — how fine-grained the gaps between adjacent representable floats are at that scale. Because of the floating-point format's own sliding exponent, this same relative gap size holds at any magnitude — which is exactly why Chapter 2's "absolute vs. relative error" distinction matters so much here.
The Four IEEE Rounding Modes
Whenever an arithmetic result doesn't fit exactly into the available mantissa bits, it has to be rounded to the nearest representable value — and IEEE 754 defines specific, precise rules for how that rounding happens. Four modes are the ones exposed by virtually every language and standard library:
| Mode | Rule | Verified example |
|---|---|---|
| Round to nearest, ties to even (the default) | Round to the closest representable value; on an exact tie, round to whichever neighbor has an even last digit | round(0.5)=0, round(1.5)=2, round(2.5)=2, round(3.5)=4 — every tie breaks toward the even number |
| Round toward zero (truncation) | Always round toward 0, discarding everything past the cutoff | trunc(2.7)=2, trunc(−2.7)=−2 |
Round toward +∞ (ceiling) | Always round up, toward positive infinity | ceil(2.3)=3, ceil(−2.3)=−2 |
Round toward −∞ (floor) | Always round down, toward negative infinity | floor(2.7)=2, floor(−2.7)=−3 |
(Honest note: the IEEE 754 standard technically also defines a fifth rounding attribute, round-ties-away-from-zero, but it's rarely exposed as a default in mainstream hardware or languages the way the four above are — round-to-nearest-ties-to-even is overwhelmingly the one that matters day to day, which is why the rest of this section focuses on it.)
Why "Ties to Even" Specifically — A Verified Demonstration
Round-half-up (the rounding most people learn in school — 0.5 always rounds to 1) seems more intuitive than round-half-even. But it has a real, measurable flaw: applied repeatedly to many values that each happen to land exactly on a .5 boundary, it introduces a systematic upward bias, because every single tie breaks the same direction.
0.5, 1.5, 2.5, ..., 999.5 (whose exact sum is 500,000.0) individually, then summing the rounded results: round-half-up gives a total of 500,500 — a bias of +500, since every single one of the 1,000 ties rounded upward. Round-half-to-even gives a total of exactly 500,000 — zero bias, because the ties alternate between rounding up and down (evenly, since consecutive integers alternate even/odd) and the errors cancel out over many roundings.
Where This Connects
| This chapter's finding | What it sets up |
|---|---|
Machine epsilon = 2⁻⁵², derived directly from the mantissa | Chapter 4's catastrophic cancellation — the precise threshold at which subtracting two close numbers destroys essentially all remaining precision |
| Relative error as the meaningful measure | Chapter 6's error propagation and conditioning, both built entirely on relative rather than absolute error |
| Per-operation rounding error compounding over many operations | Directly formalizes Chapter 1's own million-addition finding and this chapter's own 1,000-tie bias experiment into a general principle |
Hands-On Exercises
A measurement system reports a true value of 50.0 with an approximation of 50.5, and a second true value of 0.05 with an approximation of 0.055. Compute the absolute and relative error for both cases, and explain which one represents the more serious measurement problem, using this chapter's own reasoning about why relative error is usually the more meaningful measure.
Using this chapter's own halving-experiment logic, explain why the loop while 1.0 + eps != 1.0: eps /= 2 must eventually stop — that is, why continually halving eps is guaranteed to eventually produce a value small enough that adding it to 1.0 makes no difference, rather than looping forever.
A billing system needs to round exactly-half-cent amounts (values ending precisely in .5 cents) many thousands of times per day. Using this chapter's own verified 1,000-tie experiment, explain specifically why choosing round-half-up over round-half-to-even for this system would be a real, measurable design mistake rather than a harmless stylistic choice.
Chapter 3 Quick Reference
- Absolute error
= |approx − true|; relative error= |approx − true| / |true|— verified: the same absolute error (0.0001) meant a negligible10⁻¹⁰relative error in one case and a catastrophic1.0(100%) relative error in another - Machine epsilon — the smallest
εwhere1.0 + ε != 1.0— derived directly by halving:2⁻⁵² ≈ 2.22 × 10⁻¹⁶, matching Chapter 2's own 52-bit mantissa exactly - Four common IEEE rounding modes: round-to-nearest-ties-to-even (the default), round toward zero, round toward
+∞, round toward−∞ - Verified: rounding 1,000 exact
.5ties with round-half-up produces a+500systematic bias; round-half-to-even produces zero bias — the real, measured reason IEEE 754 defaults to ties-to-even - Next chapter: Catastrophic cancellation — what happens when subtraction meets two numbers that are already close to their own rounding limit