Exercise 1: Why Self-Attention Resolves "It"/"Cat" Without a Step-by-Step Relay — Possible Solution ==================================================================== HOW AN RNN WOULD HAVE TO HANDLE THIS, PER nn1-8 ------------------------------ Per nn1-8, an RNN processes a sequence step by step, carrying a compressed hidden state forward from one time step to the next — any information about "cat" that needs to influence how "it" is later processed must survive being carried through every single intervening word's own hidden-state update, one step at a time, all the way from "cat" to "it." WHAT SELF-ATTENTION DOES INSTEAD, PER THIS CHAPTER ------------------------------ Per this chapter, "self-attention lets every position in a sequence directly look at every other position simultaneously, computing a weighted combination based on how relevant each other position actually is." Per this chapter's own worked example, "self-attention lets the token 'it' attend strongly and directly to 'cat,' regardless of how many words sit between them — no need to pass that information step-by-step through every intervening word's own hidden state." WHY THIS AVOIDS THE RELAY ENTIRELY ------------------------------ Self-attention computes the relationship between "it" and "cat" as one direct calculation — comparing "it" against every other token in the sentence at once, including "cat" — rather than requiring "cat"'s own information to be repeatedly re-encoded and passed forward through "sat," "on," "the," "mat," and "because" one at a time before ever reaching "it." The distance between "it" and "cat" in the sentence has no bearing on how directly they can relate to each other under self-attention, unlike the RNN's own step-by-step relay, where every intervening word is a mandatory stop along the way. WHY THIS DIFFERENCE MATTERS SPECIFICALLY FOR LONG SEQUENCES ------------------------------ Per nn1-8's own vanishing-gradient reasoning, an RNN's step-by-step relay repeatedly multiplies the same weight's own gradient contribution across every intervening step, causing that signal to shrink exponentially the farther apart two related words are. In a short sentence like this chapter's own example, that decay might not yet be severe enough to actually lose the "it"-"cat" connection — but the same underlying mechanism becomes catastrophic in a much longer passage, where dozens or hundreds of words might separate a pronoun from its own antecedent. Because self-attention's own direct, one-step relationship between any two positions doesn't depend on the distance between them at all, it doesn't suffer this same distance-dependent degradation — "it" can attend just as directly to a "cat" mentioned 200 words earlier as to one mentioned 2 words earlier. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains precisely how an RNN would be forced to relay this information step by step, per nn1-8's own hidden-state mechanism, contrasts this with self-attention's own direct, distance-independent relationship per this chapter's own example, and explains why that independence from distance is exactly what makes the difference matter most as sequences grow long.