Exercise 3: Why a Dying ReLU Neuron Can Never Recover — Possible Solution
====================================================================

WHAT "DYING" MEANS FOR A RELU NEURON, PER THIS CHAPTER
------------------------------
Per this chapter, "a neuron whose weights push its weighted sum
permanently negative outputs exactly zero for every input, forever."
Per this chapter's own earlier definition, ReLU computes max(0, x) — so
whenever the neuron's own weighted sum (its pre-activation input) is
negative, the neuron's output is exactly 0, regardless of how negative
that sum is or what the actual input values were.

WHY ITS GRADIENT IS ALSO ZERO IN THAT SAME REGION
------------------------------
Per this chapter, "since ReLU's own gradient is also exactly zero for
negative inputs, that neuron can never recover through ordinary
gradient-based training once it lands there." ReLU's own defining shape
is flat at exactly 0 for the entire negative-input region — a flat line
has, by definition, a slope (gradient) of zero everywhere along it. So
for any input that keeps this neuron's own weighted sum negative, the
gradient computed for that neuron during backpropagation is precisely
zero, not merely small.

WHY A ZERO GRADIENT SPECIFICALLY PREVENTS RECOVERY
------------------------------
Per Exercise 1's own reasoning, backpropagation adjusts weights in
proportion to the computed gradient — a larger gradient produces a
larger weight update, a smaller gradient produces a smaller one, and a
gradient of exactly zero produces NO update at all. If a neuron's
weighted sum is negative for every training example it ever encounters,
its gradient is zero for every single one of those examples too — which
means backpropagation never has any basis to adjust that neuron's own
weights in any direction whatsoever. The neuron is stuck not because
recovery is merely slow or difficult, but because the training
mechanism itself receives literally no signal instructing it to change
anything about that neuron.

WHY THIS IS QUALITATIVELY WORSE THAN ORDINARY SATURATION
------------------------------
Unlike sigmoid or tanh's own saturation (Exercise 2), where the
gradient shrinks toward zero but is never exactly, permanently zero
everywhere the neuron might operate, a dead ReLU neuron's gradient is
EXACTLY zero across its entire negative-input region, with no
gradual tapering that might eventually provide some small, recoverable
signal. Once a neuron's weights settle into always producing a negative
weighted sum for the data it sees, there is no mathematical mechanism
within ordinary gradient descent capable of nudging it back out — it's
permanently, not just temporarily, unproductive.

WHY THIS WORKS AS AN ANSWER
------------------------------
It explains precisely what a dying ReLU neuron's own output and
gradient look like (0 and exactly 0, respectively, for negative
weighted sums), and connects this directly to how backpropagation's own
weight-update mechanism (proportional to gradient) becomes powerless to
change anything about a neuron whose gradient is permanently zero for
every input it ever sees.