Exercise 2: What Saturation Means, and Why Zero-Centering Doesn't Fix It — Possible Solution
====================================================================

WHAT SATURATION MEANS, PER THIS CHAPTER
------------------------------
Per this chapter, "for very large positive or very large negative
inputs, sigmoid's own curve goes nearly flat — its gradient (slope)
approaches zero in those regions, a state called saturation. A neuron
whose weighted sum regularly lands far out in either flat tail produces
almost no useful gradient." A saturated neuron is one whose input has
pushed it so far toward one extreme of its own output range that
further changes to its input barely move its output at all — the curve
has essentially flattened out, so the gradient (which measures exactly
how much output changes per unit change in input) shrinks toward zero.

WHY THIS IS A REAL PROBLEM FOR TRAINING
------------------------------
Per Exercise 1's own reasoning, backpropagation depends on meaningful
gradients to know how to adjust weights. A saturated neuron's own
near-zero gradient means backpropagation receives almost no signal
about how to improve that neuron's own weights, even if that neuron is
currently producing a badly wrong output — the same fundamental problem
the step function had, just less extreme and confined to specific
input ranges rather than nearly everywhere.

WHAT TANH'S ZERO-CENTERING ACTUALLY FIXES
------------------------------
Per this chapter, "sigmoid's outputs are always positive, which can
bias how gradients accumulate across a whole layer in later training
steps; tanh's own (-1, 1) range avoids that specific bias." Because
every sigmoid output is strictly positive, gradients flowing backward
through a layer of sigmoid neurons tend to consistently push all of
that layer's own incoming weights in a related, correlated direction —
a real, separate inefficiency in how training updates accumulate,
distinct from saturation itself. Tanh's output being centered around
zero (able to be positive OR negative) avoids this specific directional
bias.

WHY THIS IS A DIFFERENT PROBLEM FROM SATURATION ITSELF
------------------------------
Zero-centering addresses the DIRECTION and CONSISTENCY of gradient
updates across a layer — a real, but separate, issue from whether the
gradient's own MAGNITUDE shrinks toward zero at extreme inputs. Per this
chapter, "tanh still saturates at its own extremes — the same
flattening, the same near-zero gradient far from center. Recentering
the output doesn't remove the underlying saturation problem, just one
specific side effect of it." Tanh's own curve, like sigmoid's, still
flattens out for sufficiently large positive or negative inputs — a
tanh neuron pushed far enough toward either of its own extremes suffers
the identical near-zero-gradient problem sigmoid does, regardless of
the fact that its output range happens to be centered differently.

WHY THIS WORKS AS AN ANSWER
------------------------------
It defines saturation precisely using the chapter's own wording,
explains the specific, separate bias problem zero-centering solves
(consistent gradient direction across a layer), and explains why that
fix is genuinely unrelated to — and therefore doesn't resolve — the
magnitude-shrinking saturation problem both sigmoid and tanh still
share at their own respective extremes.