Exercise 3: When Is Random Initialization Reliable vs. Genuinely Risky? — Possible Solution ==================================================================== WHAT MADE THIS CHAPTER'S OWN STEP 6 RELIABLE ------------------------------ This chapter verified directly that five wildly different starting points - including two genuinely extreme ones, (100,-50) and (-20,30) - all converged to essentially the same answer. This happened because the MSE loss function for linear regression is convex: per Chapter 7's own guarantee, a convex function has at most one local minimum, and that minimum is automatically the global one. There was never any other basin for the search to fall into, no matter where it started. WHAT MADE CHAPTER 7'S OWN EXAMPLE UNRELIABLE ------------------------------ Chapter 7's own quartic function was NOT convex everywhere - it had two genuinely different local minima (f=-4.416 and f=-20.236) separated by a local maximum, and which one gradient descent found depended entirely on which side of that maximum the search started. Two starting points just one unit apart landed in completely different basins there. THE ACTUAL CONDITION THAT DETERMINES WHICH SITUATION YOU'RE IN ------------------------------ An engineer can be confident that training from a random starting point will reliably reach the best possible answer specifically when the loss function is convex (or can be verified/argued to be convex, the way linear regression's MSE loss provably is). This guarantee has nothing to do with luck or how many training runs happen to work out - it's a mathematical property of the function itself, verified directly in this chapter by checking multiple starting points and confirming they agree. Trying several different random starting points and keeping the best result becomes a genuinely necessary strategy specifically when the loss function is NOT known to be convex - which describes the loss surface of essentially every real neural network, since they are built from many nested, non-linear layers (the composition Chapter 8 covered) rather than the simple, provably convex linear model this capstone used. Without a convexity guarantee, there is no way to know in advance whether a single training run landed in the best available basin or a mediocre one, so running several attempts and comparing results becomes the practical substitute for a mathematical guarantee that doesn't exist for that function. RESULT ------------------------------ The deciding factor is convexity, not the algorithm, not the starting point, and not luck: convex loss functions (like this capstone's own linear regression) make gradient descent reliable from any start; non-convex loss functions (like most real neural networks) genuinely require strategies beyond a single gradient descent run to have confidence in the final result. WHY THIS WORKS AS AN ANSWER ------------------------------ The explanation directly contrasts this chapter's own verified convex result against Chapter 7's own verified non-convex result, identifies convexity specifically (not any other factor) as the deciding condition, and connects the non-convex case honestly to real neural networks' own layered, composed structure from Chapter 8, rather than treating the two situations as arbitrary or unrelated to each other.