Exercise 3: Why Positional Encoding Is Flagged Honestly, and Why RNNs Never Needed It — Possible Solution ==================================================================== WHAT THIS CHAPTER SAYS ABOUT THE GAP DIRECTLY ------------------------------ Per this chapter, "self-attention, by itself, has no inherent notion of sequence order at all — computing how much 'it' attends to 'cat' doesn't naturally encode whether 'cat' came before or after 'it' in the sentence." This is stated plainly as a real limitation of the mechanism this chapter just introduced, not smoothed over or left implicit. WHY AN RNN NEVER FACED THIS PROBLEM AT ALL, PER nn1-8's OWN MECHANISM ------------------------------ Per nn1-8, an RNN processes a sequence "step by step," updating its own hidden state one token at a time, strictly in the order the tokens actually appear. The very act of processing token 1, then token 2, then token 3, and so on, IS the mechanism by which order gets encoded — the hidden state at any point necessarily reflects everything that came before it, in the exact order it came, simply because of how the computation unfolds over time. Order was never something an RNN had to separately represent — it was built directly into the sequential nature of the computation itself. WHY SELF-ATTENTION LACKS THIS SAME AUTOMATIC PROPERTY ------------------------------ Per this chapter, self-attention "lets every position in a sequence directly look at every other position simultaneously." Because every position is compared against every other position all at once, in parallel, rather than being processed one after another in sequence, there's no equivalent "processing order" for sequence order to be implicitly derived from — the calculation comparing "it" to "cat" is structurally identical regardless of whether "cat" appeared before or after "it" in the original sentence, unless order information is explicitly supplied as part of the input somehow. WHY THIS GAP IS WORTH FLAGGING HONESTLY RATHER THAN GLOSSING OVER ------------------------------ This is a real, structural cost of the very feature (parallel, distance-independent computation) that makes self-attention so valuable, per this chapter's own compare-table reasoning. Presenting self-attention as a strictly superior replacement for RNNs without naming this specific trade-off would be an overclaim — the mechanism gains parallelizability and structural long-range-dependency handling, but genuinely loses the "order is free" property that RNNs' own sequential processing provided automatically. Naming this honestly, rather than only listing self-attention's advantages, matches this course's own broader pattern of presenting genuine trade-offs rather than one-sided wins. WHY THIS MOTIVATES POSITIONAL ENCODING AS A SEPARATE, NECESSARY ADDITION ------------------------------ Because order information is entirely absent from the raw self- attention calculation itself, per this chapter, "a transformer needs a separate, explicit mechanism to inject position information back in — positional encoding." This isn't an optional refinement; it's a necessary supplement specifically because self-attention's own core mechanism structurally cannot represent order on its own, unlike an RNN, where order was never missing in the first place. WHY THIS WORKS AS AN ANSWER ------------------------------ It identifies precisely why RNNs never needed a separate order mechanism (order is inherent to sequential, step-by-step processing), explains why self-attention's own parallel, all-at-once computation structurally lacks that same property, and explains why honestly naming this trade-off — rather than presenting self-attention as an unambiguous improvement — matches this course's own consistent practice of stating real limitations alongside real advantages.