Capstone — Optimizing a Function From Scratch
Calculus & Optimization
Chapter 10 · Capstone — Optimizing a Function From Scratch
One continuous project: fitting a straight line to real data using nothing but gradient descent, built entirely from scratch — the exact same procedure, at genuinely tiny scale, that trains real machine learning models on real datasets.
| Step | Task | Chapter(s) used |
|---|---|---|
| 1 | Define the model and the loss function | Ch.3-4 |
| 2 | Derive the gradients via the chain rule | Ch.3, Ch.5, Ch.8 |
| 3 | Verify the gradients numerically | Ch.4 |
| 4 | Run gradient descent to convergence | Ch.6 |
| 5 | Cross-check against the exact closed-form answer | Ch.9-style verification discipline |
| 6 | Confirm convexity: five starting points, one answer | Ch.7 |
Step 1 — The Model and the Loss Function
Five data points: (1,3), (2,5), (3,7), (4,8), (5,11). A linear model, ŷ = mx+b, and Mean Squared Error loss: L(m,b) = (1/n)·Σ(mxᵢ+b−yᵢ)² — exactly the same squared-error shape Chapter 8's own network used, now averaged across many data points instead of one.
Step 2 — Deriving the Gradients via the Chain Rule
Exactly Chapter 8's own derivation pattern: an outer square wrapped around an inner linear function, differentiated per data point and summed. ∂L/∂m = (2/n)·Σ(mxᵢ+b−yᵢ)·xᵢ. ∂L/∂b = (2/n)·Σ(mxᵢ+b−yᵢ).
Step 3 — Verified Against Numerical Differentiation
∂L/∂m = −23.4, ∂L/∂b = −6.6. Numerical (central difference, h=1e−6): −23.400000, −6.600000 — matching to 8 decimal places, the exact same cross-check discipline Chapter 8 used on its own network.
Step 4 — Gradient Descent, Run From Scratch
Starting at (m,b)=(0,0), learning rate α=0.01, 2000 iterations:
iter 0: m=0.484, b=0.136, loss=31.32 → iter 10: m=1.937, b=0.558, loss=0.33 → iter 100: m=2.005, b=0.722, loss=0.17 → iter 1000: m=1.905, b=1.082, loss=0.1401 → iter 1999: m=1.900, b=1.099, loss=0.1400. Converged.
Step 5 — Cross-Checked Against the Exact Answer
m=1.9, b=1.1 exactly, loss 0.14. Gradient descent's own from-scratch result after 2000 iterations: m=1.900169, b=1.099391 — differing by only 0.0002 and 0.0006 respectively. An iterative search, knowing nothing about calculus's own closed-form shortcuts, found essentially the exact right answer purely by following the gradient downhill.
Step 6 — Confirming Convexity: The Direct Opposite of Chapter 7
Chapter 7 showed gradient descent landing in genuinely different places depending on where it started, on a non-convex function. MSE loss for linear regression is provably convex — this predicts the opposite should happen here.
(0,0) → (1.900, 1.099). (10,10) → (1.898, 1.107). (−5,−5) → (1.901, 1.096). (100,−50) → (1.923, 1.017). (−20,30) → (1.890, 1.137). Every single starting point — including two genuinely extreme ones — converges toward the identical answer, m≈1.9, b≈1.1. Exactly the guarantee Chapter 7 promised for convex functions, now confirmed directly rather than assumed.
What This Course Doesn't Cover
As stated honestly back in Chapter 1: full real-analysis rigor, multivariable calculus beyond gradients, and differential equations beyond Euler's method were all named as deliberately out of scope, and stayed out of scope through all ten chapters. This capstone fit a two-parameter model; real models have millions, but the underlying mathematics — gradients, the chain rule, gradient descent, convexity — is exactly what was built here, just at a scale this course never claimed to reach.
Where This Course Connects
Linear Algebra Fundamentals' own vector and dot-product material underwrote Chapter 5's gradient directly. Algorithms & Complexity's own iterative-algorithm framing described gradient descent's own shape from the start. Boolean Algebra & Digital Logic's sigmoid-adjacent reasoning (activation functions) connects directly to Chapter 8's own network. Within this subject, a future Numerical Methods & Floating-Point Computation course would pick up directly where Chapter 1's own floating-point catastrophic-cancellation finding and Chapter 9's Euler's-method error analysis left off.
Hands-On Exercises
Using this chapter's own MSE loss formula, compute L(m=2, b=1) directly for the five data points (1,3),(2,5),(3,7),(4,8),(5,11), showing each squared-error term.
Using this chapter's own gradient formulas, compute ∂L/∂m and ∂L/∂b at m=2, b=1 for the same five data points, and use them to compute one gradient descent step with α=0.01 starting from (2,1).
Explain, using this chapter's own Step 6 findings compared against Chapter 7's own non-convex example, what specifically would need to be true about a loss function for a machine learning engineer to be confident that training from a completely random starting point will reliably reach the best possible answer, versus a case where trying several different random starting points and keeping the best result becomes a genuinely necessary strategy.
📄 View solutionChapter 10 Quick Reference
- Full worked project: model + loss (Ch.3-4) → chain-rule gradients (Ch.5, Ch.8) → numerical verification (Ch.4) → gradient descent to convergence (Ch.6) → cross-checked against the exact closed-form answer → convexity confirmed via five starting points (Ch.7)
- Gradient descent from scratch matched the exact least-squares solution to within
0.0006 - Five wildly different starting points — including two genuinely extreme ones — all converged to the same answer, confirming Chapter 7's own convexity guarantee directly
- Out of scope: full real-analysis rigor, multivariable calculus beyond gradients, differential equations beyond Euler's method
- Course complete — Calculus & Optimization, 10 chapters, from the limit definition of a derivative to a working, verified, from-scratch model fit