Exercise 1: Why Feeding "Dog, Bites, Man" vs. "Man, Bites, Dog" Produces Different Final States — Possible Solution ==================================================================== nn1-8's OWN HIDDEN-STATE MECHANISM ------------------------------ Per nn1-8, an RNN/LSTM "maintains a hidden state — a vector updated at every step of the sequence. At each time step, the network combines the current input with the previous hidden state to compute a new one." Crucially, this means each new hidden state is a function of TWO things: the input at this exact step, AND whatever hidden state already existed going into this step (itself shaped by every earlier step). WORKING THROUGH "dog bites man" ------------------------------ Step 1: the network starts from some initial hidden state and combines it with the embedding for "dog," producing a new hidden state — call it H1 — that reflects having just seen "dog" first. Step 2: the network combines H1 with the embedding for "bites," producing H2 — a state reflecting "dog" followed by "bites," in that order. Step 3: the network combines H2 with the embedding for "man," producing the final hidden state, H3 — a state reflecting the specific sequence "dog," then "bites," then "man." WORKING THROUGH "man bites dog" ------------------------------ Step 1: combines the initial hidden state with the embedding for "man," producing H1' — reflecting having just seen "man" first (a genuinely different starting point than H1 above, since "man" and "dog" are different words with different embeddings, per nlp1-5). Step 2: combines H1' with "bites," producing H2' — reflecting "man" followed by "bites." Step 3: combines H2' with "dog," producing the final hidden state H3' — reflecting the sequence "man," then "bites," then "dog." WHY H3 AND H3' ARE GENUINELY DIFFERENT ------------------------------ Because each step's own computation depends on the PREVIOUS hidden state, and the previous hidden states diverge from the very first step (H1 reflects "dog" first, H1' reflects "man" first), every subsequent step in the two sequences is combining a genuinely different accumulated history with a genuinely different next input at each corresponding step. There's no reason for these two entirely different chains of computation to arrive at the same final value — and per the underlying mechanism, they mathematically don't, since nothing in the process ever discards or ignores the order in which inputs were received. WHY THIS IS THE DIRECT, MECHANICAL RESOLUTION OF nlp1-4's PROBLEM ------------------------------ nlp1-4 proved bag-of-words vectors are identical regardless of order because counting has no dependency on sequence position at all. This chapter's own mechanism is the structural opposite: every single step explicitly depends on what came immediately before it, which is precisely why two different orderings of the same three words necessarily produce two different final hidden states rather than collapsing to the same result the way counting-based representations did. WHY THIS WORKS AS AN ANSWER ------------------------------ It traces the hidden-state computation step by step for both orderings using nn1-8's own described mechanism, shows exactly why the two chains diverge starting from the very first step, and connects this directly to why order-dependence is a structural, guaranteed property of the mechanism rather than a coincidental outcome.