Exercise 3: Why Self-Attention Resolves nn1-9's Bottleneck, and What It Still Leaves Unresolved — Possible Solution ==================================================================== WHAT nn1-9's OWN NAMED BOTTLENECK WAS ------------------------------ Per nn1-9, and reused throughout nlp1-6/nlp1-7/this chapter, an RNN/LSTM must process a sequence one token at a time — each hidden state depends on the hidden state immediately before it, which itself depended on the one before that, all the way back to the start of the sequence. This is a strictly serial computation: step 5 literally cannot be computed until step 4 has finished, because step 5 needs step 4's own output as an input. This is exactly the "inherently serial" limitation this chapter names directly. WHY SELF-ATTENTION DOESN'T HAVE THIS SAME DEPENDENCY CHAIN ------------------------------ Per this chapter, self-attention computes attention scores between EVERY pair of tokens in a sequence, using the same score-weight-blend mechanism introduced earlier in the chapter for decoder-to-encoder attention, but now applied within a single sequence. Critically, the score between token 1 and token 5 doesn't depend on first having computed the score between token 1 and token 2, or any other pair — every pairwise relationship can be computed independently of every other one. WHY THIS MAKES SELF-ATTENTION GENUINELY PARALLELIZABLE ------------------------------ Because none of the pairwise attention computations depend on each other's results, they can all be computed at the same time, on different processor cores or GPU threads simultaneously — something structurally impossible for an LSTM's own step-5-needs-step-4 chain. This is precisely what this chapter's own finding-box states: self- attention "computes every pairwise token relationship at once, which is why it's genuinely parallelizable in a way recurrence structurally cannot be." WHAT SELF-ATTENTION STILL DOESN'T RESOLVE: ORDER ------------------------------ Per this chapter's own closing section, self-attention has no inherent sense of token order — if two tokens were swapped, the full set of pairwise relationships computed between all tokens would be identical, since attention only ever asks "how relevant is token A to token B," never "which one came first." This is a genuinely separate problem from the parallelization bottleneck: removing recurrence solves the serial-computation problem, but recurrence was also, incidentally, the thing that gave the model any sense of sequence order at all (per nlp1-6's own step-by-step accumulation). Removing it removes that side-effect too. WHY THIS IS LEFT FOR A FULL TRANSFORMER TO ADDRESS ------------------------------ Per this chapter, a real Transformer needs positional encoding specifically to reintroduce the order information self-attention itself doesn't provide — and the full multi-head, multi-layer Transformer architecture built around self-attention is substantial enough that both are deliberately deferred to llm1's own full treatment, rather than rushed through here. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains precisely why an LSTM's own step-dependency chain is inherently serial, shows why self-attention's pairwise computations have no equivalent dependency and are therefore parallelizable, and identifies the specific, separate limitation (no inherent sense of order) that self-attention alone still leaves unresolved.