Convexity, Local vs. Global Minima & Optimization Landscapes

Calculus & Optimization

Chapter 7 · Convexity, Local vs. Global Minima & Optimization Landscapes

Chapter 6's function had exactly one minimum, and gradient descent found it reliably from any starting point, at any working learning rate. That was a special property of the function, not a general guarantee. This chapter asks the honest question: what happens when a function has more than one minimum?

Convexity: The Property That Makes Chapter 6's Guarantee Work

The second-derivative test

A function is convex where its second derivative f''(x) ≥ 0. Geometrically: the function curves upward like a bowl, never a wiggle. A function that's convex everywhere has at most one local minimum — and that minimum is automatically the global minimum. Gradient descent on a convex function is guaranteed to find it, from any starting point, with any small-enough learning rate.

Verified directly

Chapter 6's own function, f(x,y)=(x−3)²+(y+1)²+5, has second derivative 2 in each variable — constant, always positive, convex everywhere. This is why every learning rate below the threshold converged to the exact same point regardless of starting location.

A Non-Convex Function: Verified Multiple Minima

Consider f(x) = x⁴/4 − x³ − 2x² + 3x. Its derivative has three roots — three critical points.

Verified directly — three critical points, checked with the second-derivative test

x≈−1.398: f≈−4.416, f''≈10.25>0 — a local minimum. x≈0.559: f''≈−6.42<0 — a local maximum, sitting between the two minima. x≈3.838: f≈−20.236, f''≈17.17>0 — a second local minimum, and by direct comparison, the global one — nearly five times lower than the first.

Gradient Descent Genuinely Gets Stuck

Verified directly — two starting points, two completely different outcomes

Starting at x=−2 (in the shallow-minimum's own basin): gradient descent converges to x=−1.398, f=−4.416. Starting at x=5 (in the deep-minimum's own basin): converges to x=3.838, f=−20.236 — a dramatically better result, found purely because of where the search happened to begin.

Verified directly — an even sharper demonstration: two starts one unit apart

Starting at x=0 (just left of the local max at 0.559): converges to the shallow minimum, f=−4.416. Starting at x=1 — barely one unit away, on the other side of that same local max: converges to the deep, global minimum, f=−20.236. A tiny difference in starting point produced a nearly 5× difference in the final result.

Real Relevance: Why Neural Network Training Is Empirically Finicky

This is exactly why training a real neural network is genuinely sensitive to weight initialization, and why the same architecture can train to noticeably different final results depending on random seed — the loss surface of a real neural network is almost never convex, and gradient descent (or its more sophisticated variants) can land in different basins depending on where the search starts, exactly like this chapter's own verified example, just in a space with millions of dimensions instead of one.

Convexity & Landscapes in Code

def f(x): return x**4/4 - x**3 - 2*x**2 + 3*x
def fprime(x): return x**3 - 3*x**2 - 4*x + 3

def gradient_descent_1d(x0, lr, iterations):
    x = x0
    for _ in range(iterations):
        x = x - lr * fprime(x)
    return x

print(gradient_descent_1d(-2, 0.01, 200))   # -1.398 -- stuck in the shallow minimum
print(gradient_descent_1d(5, 0.01, 200))    # 3.838 -- found the deep global minimum

Hands-On Exercises

Exercise 1

Using the second-derivative test, determine whether f(x) = x⁴ is convex everywhere. Compute f''(x) and check its sign at x=−2, 0, 2.

📄 View solution

Exercise 2

Using this chapter's own verified critical points for f(x) = x⁴/4 − x³ − 2x² + 3x, explain why a starting point placed exactly at x=0.559 (the local maximum itself) is a genuinely unstable, unreliable place for gradient descent to begin — referring to what the gradient's own value is there.

📄 View solution

Exercise 3

A colleague argues "since gradient descent got stuck in a worse local minimum starting from x=−2, gradient descent is a broken algorithm and shouldn't be trusted." Using this chapter's own distinction between convex and non-convex functions, explain what's wrong with this conclusion.

📄 View solution

Chapter 7 Quick Reference

Convex: f''(x)≥0 everywhere — at most one local minimum, which is automatically global; gradient descent is guaranteed to find it
Verified: a non-convex quartic has two genuinely different local minima (f=−4.416 and f=−20.236) separated by a local maximum
Verified: gradient descent from x=−2 gets stuck in the shallow minimum; from x=5 it finds the deep global one
Verified: starting points just one unit apart (x=0 vs. x=1) land in completely different basins — a nearly 5× difference in outcome
This is exactly why real neural network training is sensitive to initialization and random seed — loss surfaces are almost never convex
Next chapter: The chain rule and backpropagation