Derivatives: Definition & Rules
Calculus & Optimization
Chapter 3 · Derivatives: Definition & Rules
Chapter 2 formalized the derivative as a limit: f'(x) = lim(h→0) (f(x+h)−f(x))/h. Recomputing that limit from scratch for every function would be unworkable. This chapter derives — not just states — the mechanical rules that make derivatives fast to compute, ending with the one rule this entire course is really building toward.
The Power Rule, Derived From the Limit Definition
For f(x) = x², applying Chapter 2's own limit definition directly:
f'(x) = lim(h→0) ((x+h)² − x²)/h = lim(h→0) (x²+2xh+h² − x²)/h = lim(h→0) (2xh+h²)/h = lim(h→0) (2x+h) = 2x. The same expansion for f(x)=x³ gives lim(h→0) (3x²h + 3xh² + h³)/h = lim(h→0)(3x² + 3xh + h²) = 3x². Both match the general power rule: d/dx[xⁿ] = n·xⁿ⁻¹.
x=3: n=2 → 6, n=3 → 27, n=4 → 108, n=5 → 405 — each matching the power rule's own prediction (n·3ⁿ⁻¹) and a numerical central-difference approximation, to within 0.0001.
The Product Rule
d/dx[f(x)·g(x)] = f'(x)g(x) + f(x)g'(x)
f(x)=x², g(x)=x³ — their product is just x⁵, so the power rule alone already predicts the derivative directly: 5x⁴. At x=2: product rule gives f'(2)g(2)+f(2)g'(2) = 4×8 + 4×12 = 32+48 = 80. Direct power rule on x⁵: 5×2⁴=80. Numerical differentiation: 80.000000. All three agree exactly.
The Quotient Rule
d/dx[f(x)/g(x)] = (f'(x)g(x) − f(x)g'(x)) / g(x)²
f(x)=x³, g(x)=x — their quotient is x², predicted directly by the power rule as 2x. At x=4: quotient rule gives (3×16×4 − 64×1)/16 = (192−64)/16 = 8. Direct power rule: 2×4=8. Numerical differentiation: 8.000000. All three agree.
The Chain Rule — the Single Most Important Rule in This Course
Every other rule in this chapter handles functions built from simple combination (sum, product, quotient). The chain rule handles functions built from composition — one function's output feeding directly into another's input — which is exactly the shape of every layer of a neural network feeding into the next. Chapter 8's entire subject, backpropagation, is nothing more than this one rule, applied repeatedly.
h(x) = f(g(x)): h'(x) = f'(g(x)) · g'(x) — the derivative of the "outer" function, evaluated at the inner function's own value, multiplied by the derivative of the "inner" function.
h(x) = (x²+1)³ — an outer function f(u)=u³ wrapped around an inner function g(x)=x²+1. Chain rule: h'(x) = 3(x²+1)² · 2x. At x=2: 3×5²×4 = 3×25×4 = 300. Numerical differentiation at x=2: 300.000000. Exact match.
Rules at a Glance
| Rule | Formula |
|---|---|
| Power | d/dx[xⁿ] = nxⁿ⁻¹ |
| Product | d/dx[fg] = f'g + fg' |
| Quotient | d/dx[f/g] = (f'g − fg')/g² |
| Chain | d/dx[f(g(x))] = f'(g(x))·g'(x) |
Derivative Rules in Code
Hands-On Exercises
Using the power rule, find the derivative of f(x) = x⁴. Then, using the product rule on f(x) = x·x³ (the same function, written as a product), confirm you get the identical result at x=2.
Using the chain rule, find the derivative of h(x) = (3x+1)², and evaluate it at x=1. Then verify your answer using numerical differentiation (central difference, h=1e-6) at the same point.
Explain, using this chapter's own framing, why the chain rule specifically — rather than the product or quotient rule — is the rule most directly connected to how a neural network computes its own gradients.
📄 View solutionChapter 3 Quick Reference
- Power rule:
d/dx[xⁿ] = nxⁿ⁻¹— derived directly from Chapter 2's own limit definition, not just stated - Product rule:
f'g + fg'. Quotient rule:(f'g − fg')/g²— both cross-verified against direct power-rule results and numerical differentiation - Chain rule:
f'(g(x))·g'(x)— for composed functions, verified numerically to match exactly - The chain rule is the single most important rule in this course — Chapter 8's backpropagation is this rule, applied repeatedly through a computational graph
- Next chapter: Derivatives in practice — common functions and real interpretation