The Chain Rule & Backpropagation

Calculus & Optimization

Chapter 8 · The Chain Rule & Backpropagation

Chapter 3 promised this chapter directly: the chain rule is the single most important rule in this course because backpropagation — the algorithm that trains every neural network — is nothing more than that one rule, applied repeatedly. This chapter delivers on that promise in full, on a real, if tiny, network.

A Genuine Tiny Neural Network

The computational graph

z1 = w1·x + b1 (a linear layer) → a1 = σ(z1) (Chapter 4's sigmoid activation) → z2 = w2·a1 + b2 (a second linear layer) → L = (z2 − y_target)² (squared-error loss). Four learnable parameters — w1, b1, w2, b2 — and training means finding the values that make L as small as possible, exactly Chapter 6's own optimization problem.

A concrete forward pass, verified directly

With x=1.5, w1=0.5, b1=0.2, w2=−0.8, b2=0.3, y_target=1.0: z1=0.95, a1=σ(0.95)=0.7211, z2=−0.2769, L=(−0.2769−1.0)²=1.6305.

The Backward Pass: The Chain Rule, Applied Four Times

To run gradient descent, every one of the four parameters needs its own gradient — ∂L/∂w1, ∂L/∂b1, ∂L/∂w2, ∂L/∂b2. Each is a chain of derivatives, computed backward from the loss:

Every link in the chain, computed explicitly

∂L/∂z2 = 2(z2−y_target) = −2.5538 (squared-error derivative, Chapter 3). ∂z2/∂a1 = w2 = −0.8. ∂a1/∂z1 = a1(1−a1) = 0.2011 (Chapter 4's own sigmoid derivative, reused directly). ∂z1/∂w1 = x = 1.5. Chaining: ∂L/∂w1 = ∂L/∂z2 · ∂z2/∂a1 · ∂a1/∂z1 · ∂z1/∂w1 = 0.6163. The same chain, stopping one link earlier (∂z1/∂b1=1 instead of x), gives ∂L/∂b1 = 0.4109. The output layer needs a shorter chain: ∂L/∂w2 = ∂L/∂z2 · a1 = −1.8416, ∂L/∂b2 = ∂L/∂z2 = −2.5538.

Why this is exactly backpropagation

Notice ∂L/∂z2 was computed once and reused for every downstream gradient — the "error signal" flows backward from the loss, layer by layer, each layer's own local derivative getting multiplied in along the way. That reuse is precisely what makes backpropagation efficient rather than recomputing the whole chain from scratch for every single parameter.

Verified Against the Whole Network, End to End

Every gradient matches numerical differentiation exactly

Nudging each parameter by ±1e−6 and re-running the entire forward pass, ignoring the chain-rule derivation completely: ∂L/∂w1 numerically =0.616304, chain rule =0.616304. ∂L/∂b1: 0.410869 both ways. ∂L/∂w2: −1.841573 both ways. ∂L/∂b2: −2.553784 both ways. Every single one matches exactly — the chain rule, applied by hand through four composed functions, produces identically the same answer as brute-force perturbing the whole network.

Closing the Loop: Actually Training This Network

Feeding these exact gradients into Chapter 6's own gradient descent update, learning rate α=0.5:

Verified directly — the loss genuinely falls

step 0: loss=1.630454 → step 1: 0.416999 → step 2: 0.042482 → step 3: 0.004199 → step 4: 0.000381 → step 5: 0.000035 → after 6 total updates: loss=0.000003. Every piece of this course — the chain rule (Ch.3), the gradient (Ch.5), gradient descent (Ch.6) — working together on a real, if miniature, trained network.

Backpropagation in Code

import math
def sigmoid(z): return 1/(1+math.exp(-z))

def forward(x, w1, b1, w2, b2, y):
    z1 = w1*x + b1
    a1 = sigmoid(z1)
    z2 = w2*a1 + b2
    L = (z2 - y)**2
    return z1, a1, z2, L

def backward(x, w1, b1, w2, b2, y):
    z1, a1, z2, L = forward(x, w1, b1, w2, b2, y)
    dL_dz2 = 2*(z2 - y)
    da1_dz1 = a1*(1-a1)                 # Chapter 4's sigmoid derivative
    dL_dw1 = dL_dz2 * w2 * da1_dz1 * x   # chain rule, 4 links
    dL_db1 = dL_dz2 * w2 * da1_dz1 * 1
    dL_dw2 = dL_dz2 * a1                 # chain rule, 2 links
    dL_db2 = dL_dz2
    return dL_dw1, dL_db1, dL_dw2, dL_db2, L

# train for 6 steps
x, w1, b1, w2, b2, y = 1.5, 0.5, 0.2, -0.8, 0.3, 1.0
for _ in range(6):
    dw1, db1, dw2, db2, L = backward(x, w1, b1, w2, b2, y)
    print(L)
    w1, b1, w2, b2 = w1-0.5*dw1, b1-0.5*db1, w2-0.5*dw2, b2-0.5*db2

Hands-On Exercises

Exercise 1

Using this chapter's own computational graph and the values z2=−0.2769, y_target=1.0, compute ∂L/∂z2 directly, showing each part of the calculation.

📄 View solution

Exercise 2

Using this chapter's own chain — ∂L/∂z2=−2.5538, w2=−0.8, a1=0.7211 — compute ∂a1/∂z1 using Chapter 4's own sigmoid derivative formula, then assemble the full chain rule product for ∂L/∂b1 (using ∂z1/∂b1=1), showing each link multiplied in.

📄 View solution

Exercise 3

Explain, using this chapter's own worked example, specifically why ∂L/∂w2's own chain (∂L/∂z2 · ∂z2/∂w2) is shorter than ∂L/∂w1's own chain (four links). What does this tell you about how the length of the chain rule needed for a parameter's gradient relates to that parameter's own position in the network?

📄 View solution

Chapter 8 Quick Reference

Backpropagation is the chain rule (Chapter 3), applied repeatedly through a network's own layers, backward from the loss
The "error signal" (∂L/∂z2 here) is computed once and reused for every downstream gradient — the actual source of backpropagation's efficiency
All four gradients verified to match numerical differentiation of the entire network exactly
Chapter 4's own sigmoid derivative, σ(x)(1−σ(x)), is reused directly as one link in the chain
Verified end to end: feeding these gradients into Chapter 6's gradient descent drove a real loss from 1.63 to 0.000003 in six steps
Next chapter: Integrals and numerical integration