Linear Algebra Fundamentals built vectors. Algorithms & Complexity built the idea of a repeated, improving loop. This course fuses them: a gradient is a vector (Chapter 5), and gradient descent (Chapter 6) is exactly the kind of repeated-improvement loop Algorithms & Complexity already covered — applied to a genuinely enormous real payoff: it's the literal algorithm that trains every machine learning model in production use today.

What a Derivative Actually Measures

A derivative is the instantaneous rate of change of a function — how fast its output moves as its input moves, at one exact point. Geometrically, it's the slope of the line tangent to the function's curve at that point. Practically, for a programmer, it answers one specific and enormously useful question: if I nudge this input slightly, which direction does the output move, and how fast?

A Real Demonstration: Derivatives Are Computable — With a Real Catch

For f(x) = x², the true derivative is f'(x) = 2x — a fact this chapter states, not yet proves (Chapter 3 derives it properly). But a derivative can also be approximated numerically, with no symbolic calculus at all, using the definition itself: (f(x+h) − f(x)) / h, for a small step h.

Five Concrete Connections to Real Code

What This Course Won't Cover

Calculus as a full field is enormous, and this course deliberately covers only what's needed to understand and implement real optimization:

• Full real-analysis rigor — formal epsilon-delta limit proofs and the deeper theory behind why calculus works stay out of scope; this course treats limits and derivatives practically, not axiomatically
• Multivariable calculus beyond gradients — Hessians, Jacobians, vector calculus (divergence, curl) are genuinely useful in specialized contexts but aren't needed for gradient descent itself, this course's own central destination
• Differential equations — beyond the basic numerical integration (Euler's method) covered in Chapter 9, solving differential equations analytically is its own substantial topic

Where This Course Is Headed

Hands-On Exercises

Chapter 1's numerical experiment shrank h toward zero and watched the approximation "converge toward" the true derivative — without ever setting h exactly to zero. That word, converge, has a precise name: a limit. This chapter formalizes it, practically rather than with full epsilon-delta rigor, per Chapter 1's own honest scope note.

What a Limit Actually Captures

Why This Matters for Derivatives Specifically

The derivative is defined as a limit: f'(x) = lim(h→0) (f(x+h)−f(x))/h. Plugging in h=0 directly gives 0/0 — genuinely undefined, a real division by zero. The limit is precisely the tool that lets Chapter 1's whole numerical experiment make sense: reasoning about what the expression approaches as h shrinks toward zero, without ever actually dividing by zero.

A Worked Example: A Limit That Exists Where the Function Doesn't

Consider f(x) = (x²−4)/(x−2). Substituting x=2 directly gives 0/0 — undefined, a genuine division-by-zero error.

Continuity: When the Limit and the Function Actually Agree

A Real Case: Continuous, But Not Differentiable

Continuity is necessary for differentiability, but not sufficient — a function can be perfectly continuous and still have no well-defined derivative at a point. f(x) = |x| at x=0 is the classic case.

Real Relevance

Chapter 6's gradient descent assumes it can always ask "which direction does this function decrease in?" — a question that only has a clean answer where the function is differentiable. Functions with sharp corners (like ReLU, an extremely common neural-network activation function, which is literally max(0,x) — a close cousin of |x|) genuinely have points where the derivative doesn't exist in the classical sense, a real practical wrinkle real ML libraries handle by defining a sensible value at that single point rather than pretending the problem doesn't exist.

Limits in Code

Hands-On Exercises

Chapter 2 formalized the derivative as a limit: f'(x) = lim(h→0) (f(x+h)−f(x))/h. Recomputing that limit from scratch for every function would be unworkable. This chapter derives — not just states — the mechanical rules that make derivatives fast to compute, ending with the one rule this entire course is really building toward.

The Power Rule, Derived From the Limit Definition

For f(x) = x², applying Chapter 2's own limit definition directly:

The Product Rule

The Quotient Rule

The Chain Rule — the Single Most Important Rule in This Course

Every other rule in this chapter handles functions built from simple combination (sum, product, quotient). The chain rule handles functions built from composition — one function's output feeding directly into another's input — which is exactly the shape of every layer of a neural network feeding into the next. Chapter 8's entire subject, backpropagation, is nothing more than this one rule, applied repeatedly.

Rules at a Glance

Derivative Rules in Code

Hands-On Exercises

Chapter 3 built the rules. This chapter applies them to the specific functions that show up constantly in real code — exponentials, logarithms, and trig functions — and formalizes the numerical differentiation technique this course has been using informally since Chapter 1 into a proper, comparable tool.

Common Function Derivatives, Verified

Real Relevance: The Sigmoid Function's Own Derivative

The sigmoid function, σ(x) = 1/(1+e⁻ˣ), is one of the most common activation functions in machine learning — built entirely from e^x. Applying the chain and quotient rules (Chapter 3) gives it an unusually elegant derivative:

What the Sign and Magnitude of a Derivative Actually Mean

Numerical Differentiation, Formalized: Forward vs. Central Difference

Every prior chapter used the difference quotient informally. There are actually two natural versions: forward difference, (f(x+h)−f(x))/h, and central difference, (f(x+h)−f(x−h))/(2h) — averaging a step forward and a step back.

Common Derivatives in Code

Hands-On Exercises

Every derivative so far has tracked one variable. A real loss function has as many variables as a model has parameters — sometimes millions. This chapter extends derivatives to that world, and lands on the exact object Chapter 6 needs: a vector, built directly from Linear Algebra Fundamentals' own material, that tells gradient descent which way to move.

Partial Derivatives: One Variable at a Time

For f(x,y) = x²y + y³:

The Gradient: A Vector of Partial Derivatives

A Full, Verified Demonstration: The Gradient Points in the Direction of Steepest Ascent

For f(x,y) = x²+y² at the point (1,2): f(1,2)=5, and the gradient is ∇f=(2x,2y)=(2,4).

Why: the Directional Derivative Is a Dot Product

The rate of change of f in any unit direction u is ∇f · u — a genuine dot product, straight from Linear Algebra Fundamentals. Since a dot product is maximized exactly when two vectors point the same way, the direction that maximizes ∇f · u is u = ∇f itself — which is why the gradient is the direction of steepest ascent, not merely an observed pattern.

Real Relevance

This is the entire reason Chapter 6's gradient descent works at all: at any point, the negative gradient — −∇f — points in the direction of steepest decrease, exactly the direction needed to minimize a loss function. Every parameter update in every gradient-descent-trained model is, at its core, exactly the demonstration verified above, just run in reverse and repeated many times.

Partial Derivatives & Gradients in Code

Hands-On Exercises

Chapter 5 proved the negative gradient points toward steepest decrease. This chapter turns that single geometric fact into a repeatable algorithm — the literal procedure that trains linear regression, logistic regression, and every deep neural network in production use.

The Algorithm

A Fully Traced, Verified Minimization

Minimizing f(x,y) = (x−3)² + (y+1)² + 5 — true minimum at (3,−1), value 5 — starting from (0,0), learning rate α=0.1:

The Learning Rate: Five Regimes, All Verified

The same starting point, the same function, five different learning rates — the outcome changes completely.

Why: the Exact Convergence Threshold

For this specific function, each gradient descent step is a simple linear update: x_new = x·(1−2α) + 6α. Whether this converges depends entirely on |1−2α|.

Gradient Descent in Code

Hands-On Exercises

Chapter 6's function had exactly one minimum, and gradient descent found it reliably from any starting point, at any working learning rate. That was a special property of the function, not a general guarantee. This chapter asks the honest question: what happens when a function has more than one minimum?

Convexity: The Property That Makes Chapter 6's Guarantee Work

A Non-Convex Function: Verified Multiple Minima

Consider f(x) = x⁴/4 − x³ − 2x² + 3x. Its derivative has three roots — three critical points.

Gradient Descent Genuinely Gets Stuck

Real Relevance: Why Neural Network Training Is Empirically Finicky

This is exactly why training a real neural network is genuinely sensitive to weight initialization, and why the same architecture can train to noticeably different final results depending on random seed — the loss surface of a real neural network is almost never convex, and gradient descent (or its more sophisticated variants) can land in different basins depending on where the search starts, exactly like this chapter's own verified example, just in a space with millions of dimensions instead of one.

Convexity & Landscapes in Code

Hands-On Exercises

Chapter 3 promised this chapter directly: the chain rule is the single most important rule in this course because backpropagation — the algorithm that trains every neural network — is nothing more than that one rule, applied repeatedly. This chapter delivers on that promise in full, on a real, if tiny, network.

A Genuine Tiny Neural Network

The Backward Pass: The Chain Rule, Applied Four Times

To run gradient descent, every one of the four parameters needs its own gradient — ∂L/∂w1, ∂L/∂b1, ∂L/∂w2, ∂L/∂b2. Each is a chain of derivatives, computed backward from the loss:

Verified Against the Whole Network, End to End

Closing the Loop: Actually Training This Network

Feeding these exact gradients into Chapter 6's own gradient descent update, learning rate α=0.5:

Backpropagation in Code

Hands-On Exercises

Chapters 3-8 built one half of calculus: given a function, find its rate of change. This chapter builds the other half — given a rate of change, find the accumulated total — and lands on the specific technique real physics engines and simulations run every single frame.

The Integral as Accumulated Area

Numerical Integration: Four Methods, Compared

Real functions in code (or genuinely unknown functions from sensor data) don't come with a symbolic antiderivative. Numerical integration approximates the area directly, by summing many small pieces.

Euler's Method: Numerical Integration for Physics Simulation

If acceleration is known, integrating once gives velocity; integrating again gives position. A physics engine does this numerically, one small time step dt at a time — Euler's method: v_new = v + a·dt, x_new = x + v·dt.

Numerical Integration in Code

Hands-On Exercises

One continuous project: fitting a straight line to real data using nothing but gradient descent, built entirely from scratch — the exact same procedure, at genuinely tiny scale, that trains real machine learning models on real datasets.

Step 1 — The Model and the Loss Function

Step 2 — Deriving the Gradients via the Chain Rule

Step 3 — Verified Against Numerical Differentiation

Step 4 — Gradient Descent, Run From Scratch

Step 5 — Cross-Checked Against the Exact Answer

Step 6 — Confirming Convexity: The Direct Opposite of Chapter 7

What This Course Doesn't Cover

As stated honestly back in Chapter 1: full real-analysis rigor, multivariable calculus beyond gradients, and differential equations beyond Euler's method were all named as deliberately out of scope, and stayed out of scope through all ten chapters. This capstone fit a two-parameter model; real models have millions, but the underlying mathematics — gradients, the chain rule, gradient descent, convexity — is exactly what was built here, just at a scale this course never claimed to reach.

Where This Course Connects

Linear Algebra Fundamentals' own vector and dot-product material underwrote Chapter 5's gradient directly. Algorithms & Complexity's own iterative-algorithm framing described gradient descent's own shape from the start. Boolean Algebra & Digital Logic's sigmoid-adjacent reasoning (activation functions) connects directly to Chapter 8's own network. Within this subject, a future Numerical Methods & Floating-Point Computation course would pick up directly where Chapter 1's own floating-point catastrophic-cancellation finding and Chapter 9's Euler's-method error analysis left off.

Hands-On Exercises