Exercise 3: Why LSTMs Fixed Vanishing Gradients But Not Sequential Computation — Possible Solution ==================================================================== WHAT LSTMS ACTUALLY FIX, PER THIS CHAPTER ------------------------------ Per this chapter, LSTM gates "let the network learn when to preserve information essentially unchanged and when to actually update it. This provides a much more direct path for gradients to flow backward through the cell state across many time steps, largely avoiding the repeated same-weight-multiplied-by-itself collapse plain RNNs suffer from." This directly addresses the specific mathematical problem Exercise 2 examined — the exponential shrinkage caused by the same weight factor being multiplied by itself over and over across a long sequence. The gated cell state provides an alternative path where information (and gradients) can pass through largely unmodified rather than being forced through that repeated multiplicative shrinkage every single step. WHAT LSTMS DO NOT CHANGE, PER THIS CHAPTER ------------------------------ Per this chapter, "an RNN (LSTM or not) is inherently sequential — each time step's own computation depends on the previous step's own output, which means... the steps within one single sequence can't be computed simultaneously." An LSTM cell, exactly like a plain RNN cell, still needs the previous time step's own cell state and hidden state as direct inputs before it can compute the current time step's own output — nothing about adding gates changes this fundamental dependency chain. Step 5 still cannot begin its own computation until step 4 has actually finished, regardless of how sophisticated the internal gating mechanism at each individual step has become. WHY THESE ARE GENUINELY TWO SEPARATE PROBLEMS ------------------------------ The vanishing-gradient problem is about whether USEFUL GRADIENT INFORMATION survives the journey backward across many time steps during TRAINING — a question about how well information propagates through the network's own computation once the full sequence has already been processed. The sequential-computation problem is about whether the FORWARD computation across time steps can be PARALLELIZED — a question about the basic structural dependency between consecutive steps, entirely separate from whether gradients flow well or poorly once that computation is done. A network could, in principle, have perfect gradient flow (no vanishing at all) while still being forced to compute every time step one after another, in strict sequence — exactly LSTMs' own actual situation. WHY FIXING ONE DOESN'T TOUCH THE OTHER ------------------------------ The gates that solve the vanishing-gradient problem operate entirely WITHIN a single time step's own internal computation (deciding how much to forget, add, and output) — they don't change the fact that this entire internal computation still requires the previous step's own output as an input before it can even begin. Adding more sophisticated internal machinery to each step doesn't remove the step-to-step dependency chain that makes the overall sequence processing inherently sequential. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains precisely what mechanism lets LSTM gates fix vanishing gradients (a more direct path for gradient flow through the cell state), explains why that same mechanism leaves the step-to-step dependency chain fully intact, and distinguishes the two problems as addressing genuinely different aspects of RNN computation — gradient flow during training vs. parallelizability of the forward computation itself.