Derivatives in Practice: Common Functions & Real Interpretation

Calculus & Optimization

Chapter 4 · Derivatives in Practice: Common Functions & Real Interpretation

Chapter 3 built the rules. This chapter applies them to the specific functions that show up constantly in real code — exponentials, logarithms, and trig functions — and formalizes the numerical differentiation technique this course has been using informally since Chapter 1 into a proper, comparable tool.

Common Function Derivatives, Verified

Function	Derivative
e^x	`e^x` — its own derivative
ln(x)	`1/x`
sin(x)	`cos(x)`
cos(x)	`−sin(x)`

Verified directly, all four

At x=1.5: e^1.5=4.481689, numerical derivative =4.481689 — identical, confirming e^x genuinely is its own derivative. At x=2: ln'(x) numerically =0.5, matching 1/x=0.5 exactly. At x=π/4: sin'(x) numerically =0.707107, matching cos(π/4)=0.707107; cos'(x) numerically =−0.707107, matching −sin(π/4)=−0.707107.

Real Relevance: The Sigmoid Function's Own Derivative

The sigmoid function, σ(x) = 1/(1+e⁻ˣ), is one of the most common activation functions in machine learning — built entirely from e^x. Applying the chain and quotient rules (Chapter 3) gives it an unusually elegant derivative:

Verified directly

σ'(x) = σ(x)·(1−σ(x)) — the derivative is expressible entirely in terms of the function's own output, needing no re-evaluation of e^x at all. At x=0.5: formula gives 0.235004, numerical differentiation gives 0.235004 — exact match. This clean form is precisely why sigmoid was historically such a popular activation function: computing its gradient during training is nearly free once the forward pass has already computed σ(x).

What the Sign and Magnitude of a Derivative Actually Mean

Derivative value	What it means, geometrically
f'(x) > 0	Function is increasing at that point
f'(x) < 0	Function is decreasing
f'(x) = 0	A critical point — a local max, local min, or saddle (Chapter 7 sorts out which)
\|f'(x)\| large	Steep — the function is changing rapidly
\|f'(x)\| small	Nearly flat — the function is barely changing

Numerical Differentiation, Formalized: Forward vs. Central Difference

Every prior chapter used the difference quotient informally. There are actually two natural versions: forward difference, (f(x+h)−f(x))/h, and central difference, (f(x+h)−f(x−h))/(2h) — averaging a step forward and a step back.

Verified directly — central difference is dramatically more accurate for the same h

For f(x)=x³ at x=3 (true derivative 27): at h=0.1, forward difference error is 0.91, central difference error is only 0.01 — 91× more accurate, using the identical step size. At h=0.01: forward error 0.09, central error 0.0001 — roughly 900× more accurate. Central difference's error shrinks roughly with the square of h, while forward difference's error shrinks only linearly with h.

Why this matters in practice

Chapter 1's own numerical experiment used a forward-style difference. Central difference reaches the same accuracy with a far larger, floating-point-safer h — directly softening the exact catastrophic-cancellation problem Chapter 1 flagged, without needing h to shrink nearly as far.

Common Derivatives in Code

import math

def forward_diff(f, x, h): return (f(x+h) - f(x)) / h
def central_diff(f, x, h): return (f(x+h) - f(x-h)) / (2*h)

def sigmoid(x): return 1 / (1 + math.exp(-x))
def sigmoid_deriv(x):
    s = sigmoid(x)
    return s * (1 - s)   # the clean closed form -- no re-computing e^x

x0 = 0.5
print(sigmoid_deriv(x0), central_diff(sigmoid, x0, 1e-6))   # match

Hands-On Exercises

Exercise 1

Using this chapter's own derivative table, find the derivative of f(x) = ln(x) at x=5, then verify it numerically using central difference with h=1e-6.

📄 View solution

Exercise 2

Compute the sigmoid function's own derivative at x=0 using this chapter's own closed-form formula σ(x)(1−σ(x)), given that σ(0)=0.5 exactly. Explain, using this chapter's own sign/magnitude table, what a derivative of this specific value means about how steeply the sigmoid function is changing at x=0 compared to at very large or very negative x (where σ(x) approaches 0 or 1).

📄 View solution

Exercise 3

For f(x)=x² at x=1 (true derivative 2), compute the forward difference and central difference approximations at h=0.5, and compare their errors. Does this example show the same "central difference is dramatically more accurate" pattern this chapter found for x³? If not, explain why not, referring to what's special about x² specifically.

📄 View solution

Chapter 4 Quick Reference

e^x is its own derivative. ln(x)' = 1/x. sin(x)' = cos(x), cos(x)' = −sin(x) — all verified numerically
Sigmoid's derivative: σ(x)(1−σ(x)) — an elegant closed form, directly why sigmoid was historically popular in ML
Sign of f'(x): increasing/decreasing. Magnitude: how steeply. Zero: a critical point (Chapter 7 sorts out which kind)
Central difference is dramatically more accurate than forward difference for the same step size — verified directly (91× to 900× more accurate)
Central difference's error shrinks with h²; forward difference's error shrinks only with h
Next chapter: Partial derivatives and the gradient