Exercise 3: Why "Close to 1.0" Is Dangerous Reasoning for a Learning Rate — Possible Solution ==================================================================== THE COLLEAGUE'S REASONING ------------------------------ That alpha=1.05 should behave similarly to alpha=1.0, since the two numbers are numerically close to each other. WHAT THIS CHAPTER ALREADY VERIFIED ABOUT alpha=1.0 AND alpha=1.1 ------------------------------ alpha=1.0 is not just "a learning rate that happens to work alright" - it is the EXACT boundary between convergence and divergence for this function. At exactly alpha=1.0, the algorithm oscillates forever, neither improving nor getting worse (f stuck at 15.0 indefinitely). alpha=1.1 - a value only 0.1 away from the boundary - was already shown to genuinely diverge, with the cost climbing every single iteration (15.0 up through 133.4 within just 8 steps). WHY "CLOSE TO 1.0" IS THE WRONG WAY TO THINK ABOUT THIS ------------------------------ The behavior of gradient descent doesn't change smoothly and gradually as alpha crosses this boundary - it changes QUALITATIVELY. Below the threshold, the algorithm converges (eventually, however slowly). At the threshold, it oscillates forever without ever improving. Above the threshold, it diverges, and this chapter's own verified numbers show that divergence can already be severe just slightly past the boundary (alpha=1.1's cost nearly tripled within 7 iterations). There is no "safely a little bit over" - crossing the threshold at all, even by a small amount, changes the outcome from "eventually reaches the minimum" to "runs away forever." RESULT ------------------------------ alpha=1.05 sits on the wrong side of this exact threshold (since the threshold for this function is precisely alpha=1.0), meaning it will diverge - not "almost converge" or "converge a bit more slowly" - it will behave qualitatively like the verified alpha=1.1 case, with cost growing without bound, not like a mild variation on a working configuration. Being numerically close to a working value is not the same as being close to correct behavior once a hard threshold like this one is involved. WHY THIS WORKS AS AN ANSWER ------------------------------ The explanation is grounded directly in this chapter's own verified findings about the specific boundary at alpha=1.0 and the specific divergence already observed at alpha=1.1, rather than a generic warning about "picking learning rates carefully," and explicitly names the qualitative (not gradual) nature of the change across the threshold as the real reason numeric closeness is misleading here.