Exercise 2: Why Vanishing Gradients Are Worse for RNNs Than for nn1-5's General Case — Possible Solution ==================================================================== WHAT nn1-5's OWN GENERAL CHAIN-RULE CASE LOOKS LIKE ------------------------------ Per this chapter's own warn-box, "nn1-5's own chain rule multiplies together DIFFERENT weights at each layer — bad luck at any one layer doesn't guarantee bad luck at every other." In an ordinary deep feedforward network, each layer has its own independently-learned set of weights, so the gradient contribution multiplied in at each layer is a genuinely different number each time — some layers might contribute factors close to 1 (little shrinkage), others might contribute smaller factors, and there's no guarantee the same unfavorable value recurs identically at every single layer. WHAT HAPPENS INSTEAD IN AN RNN, PER THIS CHAPTER ------------------------------ Per this chapter, "because the same weight is reused at every time step, backpropagation through time multiplies that same weight's own gradient contribution by itself, repeatedly, once per time step. If each individual factor is even slightly less than 1... the cumulative product shrinks toward zero exponentially fast across a long sequence." Because parameter sharing (Exercise 1's own topic) means the identical weight applies at every time step, the SAME gradient contribution factor gets multiplied into the cumulative product over and over, once per time step in the sequence. WHY REPEATING THE SAME FACTOR IS MATHEMATICALLY WORSE THAN VARYING ONES ------------------------------ If a single factor slightly below 1 (say, 0.9) is multiplied by itself repeatedly across many time steps — 0.9 raised to the power of, say, 50 time steps — the result shrinks toward zero extremely quickly (exponential decay). By contrast, a sequence of genuinely different factors across different layers, some larger and some smaller, has a real chance of partially offsetting each other, or at least not compounding the exact same shrinkage rate over and over in lockstep. Repeated multiplication of the identical sub-1 value is a mathematically harsher, more relentless form of shrinkage than multiplying together a varied set of different values, even if the varied set includes some individually smaller factors too. WHY THIS IS SPECIFICALLY A CONSEQUENCE OF PARAMETER SHARING ------------------------------ This exact difference — the same weight vs. different weights being multiplied at each step — traces directly back to this chapter's own earlier point (and Exercise 1's own topic): an RNN's own parameter sharing across time is precisely what forces the identical weight to recur in the gradient calculation at every single time step, unlike a standard deep network where each layer's own distinct weights naturally introduce variation into the sequence of multiplied factors. WHY THIS MAKES A PLAIN RNN "FORGET" DISTANT TIME STEPS ------------------------------ Per this chapter, "a plain RNN effectively 'forgets' anything from more than a handful of time steps ago, because gradients from far in the past barely survive the trip back to influence early weights at all." Because the exponential shrinkage compounds so severely over a long sequence, gradient information connecting an early time step's own weights to a much-later loss becomes vanishingly small — the network effectively receives no meaningful training signal telling it to account for anything that happened many steps in the past. WHY THIS WORKS AS AN ANSWER ------------------------------ It contrasts nn1-5's own varied-factor case against this chapter's own repeated-identical-factor case, explains mathematically why repeating the same sub-1 value produces harsher exponential shrinkage than varying factors would, and ties this directly back to parameter sharing as the specific structural cause, exactly as the chapter's own warn-box frames it.