Exercise 3: Why Shrinking h and Adding Precision Pull in Opposite Directions — Possible Solution ==================================================================== WHY THEY SEEM LIKE THE SAME KIND OF FIX AT FIRST ------------------------------ Both "use a smaller h" and "use more precision" sound like generic ways to make a numerical approximation more accurate, since both are commonly reached for whenever a result looks insufficiently precise. It's tempting to treat them as interchangeable tools for the same underlying goal. WHY THEY ACTUALLY WORK AGAINST EACH OTHER HERE ------------------------------ This chapter's own resolution of the numerical-differentiation breakdown showed the real problem is specifically catastrophic cancellation in the numerator, f(x+h) - f(x), which gets worse specifically because shrinking h makes f(x+h) converge toward f(x) - i.e., it makes the two subtraction operands MORE similar to each other, not less. That is precisely the condition that causes cancellation to destroy precision, verified directly earlier in this chapter with the (1+x)-1 experiment. So shrinking h doesn't just fail to help past a certain point - it actively makes the cancellation problem worse, by design, since a smaller h always means f(x+h) and f(x) are closer together. Meanwhile, "more precision" (e.g., moving from single to double precision, or double to extended precision) only helps by pushing machine epsilon smaller, which raises the point at which cancellation becomes total - it doesn't change the fact that shrinking h keeps approaching that boundary regardless of where it sits. Extra precision buys some extra room before the same collapse happens again at an even smaller h, but it doesn't fix the underlying mechanism, which is structural to the (f(x+h)-f(x))/h formula itself, not a limitation of any specific precision level. WHY THIS MAKES THEM "OPPOSING," NOT JUST "DIFFERENT" ------------------------------ Shrinking h is the very thing driving the two subtraction operands together, which is the direct cause of the eventual collapse. Treating "smaller h" as an unconditional accuracy improvement, while technically true for the truncation error alone (which shrinks with h), ignores that it's simultaneously making a second, opposing kind of error (cancellation error) worse. The two effects are trading off against each other rather than both improving together - there is an optimal middle value of h, not an ever-shrinking one, which is exactly why Calculus & Optimization's own experiment saw accuracy improve and then reverse rather than improving indefinitely. WHY THIS WORKS AS AN ANSWER ------------------------------ The explanation identifies the specific mechanism (shrinking h directly causes the cancellation that later chapters' fixes address) rather than treating "smaller h" and "more precision" as generically similar accuracy knobs, and correctly frames the two error sources as trading off against each other rather than one strictly dominating the other.