Partial Derivatives & the Gradient

Calculus & Optimization

Chapter 5 · Partial Derivatives & the Gradient

Every derivative so far has tracked one variable. A real loss function has as many variables as a model has parameters — sometimes millions. This chapter extends derivatives to that world, and lands on the exact object Chapter 6 needs: a vector, built directly from Linear Algebra Fundamentals' own material, that tells gradient descent which way to move.

Partial Derivatives: One Variable at a Time

The idea

A partial derivative ∂f/∂x is the derivative of a multivariable function with respect to one variable, treating every other variable as a constant — literally reusing every rule from Chapter 3, just applied to one variable while the rest sit still.

For f(x,y) = x²y + y³:

Verified directly against numerical differentiation, at (2,3)

∂f/∂x = 2xy (treating y as constant, applying the power rule to the x² part). At (2,3): formula gives 2×2×3=12; numerically holding y fixed and varying x gives 12.000000 — exact match. ∂f/∂y = x²+3y² (treating x as constant). At (2,3): formula gives 4+27=31; numerically holding x fixed gives 31.000000 — exact match.

The Gradient: A Vector of Partial Derivatives

The definition

The gradient, ∇f, is the vector formed by stacking every partial derivative together: ∇f = (∂f/∂x, ∂f/∂y, ...). This is exactly the kind of object Linear Algebra Fundamentals already built machinery for — magnitude, direction, dot products all apply directly.

A Full, Verified Demonstration: The Gradient Points in the Direction of Steepest Ascent

For f(x,y) = x²+y² at the point (1,2): f(1,2)=5, and the gradient is ∇f=(2x,2y)=(2,4).

Verified directly — testing six directions, same step size

Taking a step of 0.1 in the unit gradient direction (0.447, 0.894): f increases by 0.457 — the largest increase of every direction tested. The perpendicular direction: increase of only 0.010 — almost flat. The negative gradient direction: f decreases by 0.437 — the steepest possible decrease. Two arbitrary directions, (1,0) and (0,1): increases of 0.210 and 0.410 — both real increases, but neither beats the gradient direction's own 0.457.

Why: the Directional Derivative Is a Dot Product

The rate of change of f in any unit direction u is ∇f · u — a genuine dot product, straight from Linear Algebra Fundamentals. Since a dot product is maximized exactly when two vectors point the same way, the direction that maximizes ∇f · u is u = ∇f itself — which is why the gradient is the direction of steepest ascent, not merely an observed pattern.

Verified directly — the dot product ranks every direction correctly

∇f·u for each of the six directions tested above: gradient direction 4.472 (the largest), (0,1) direction 4.0, (1,0) direction 2.0, perpendicular direction 0.0 (exactly zero rate of change), negative gradient direction −4.472 (the most negative). The ranking matches the actual measured f-increases from the six-direction test exactly.

Real Relevance

This is the entire reason Chapter 6's gradient descent works at all: at any point, the negative gradient — −∇f — points in the direction of steepest decrease, exactly the direction needed to minimize a loss function. Every parameter update in every gradient-descent-trained model is, at its core, exactly the demonstration verified above, just run in reverse and repeated many times.

Partial Derivatives & Gradients in Code

def f(x, y): return x**2 + y**2

def gradient(f, x, y, h=1e-6):
    dfdx = (f(x+h, y) - f(x-h, y)) / (2*h)
    dfdy = (f(x, y+h) - f(x, y-h)) / (2*h)
    return (dfdx, dfdy)

import math
gx, gy = gradient(f, 1, 2)
mag = math.sqrt(gx**2 + gy**2)
unit = (gx/mag, gy/mag)
print(gx, gy, unit)   # (2.0, 4.0), (0.447, 0.894)

Hands-On Exercises

Exercise 1

For f(x,y) = 3x²y − y³, find ∂f/∂x and ∂f/∂y symbolically, then evaluate both at (1,2).

📄 View solution

Exercise 2

Using your answer to Exercise 1, write out the gradient ∇f at (1,2) as a vector, then compute its magnitude.

📄 View solution

Exercise 3

Using this chapter's own directional-derivative dot-product formula, explain why the directional derivative in the direction exactly perpendicular to the gradient is always precisely zero, for any function and any point — not just the specific example this chapter tested.

📄 View solution

Chapter 5 Quick Reference

Partial derivative: derivative with respect to one variable, holding the rest constant — reuses every Chapter 3 rule directly
Gradient: the vector of every partial derivative, ∇f = (∂f/∂x, ∂f/∂y, ...)
Verified directly: stepping in the gradient's own direction gives strictly the largest increase, among six directions tested
Directional derivative = ∇f · u — a dot product, maximized exactly when u aligns with the gradient (Linear Algebra Fundamentals)
The negative gradient points toward steepest decrease — the exact mechanism Chapter 6's gradient descent is built on
Next chapter: Gradient descent — the optimization algorithm behind machine learning