Exercise 1: Why One Attention Computation Is a Real Constraint, and How Multi-Head Attention Fixes It — Possible Solution ==================================================================== WHAT A SINGLE ATTENTION COMPUTATION HAS TO REPRESENT ------------------------------ Per llm1-4, one self-attention computation produces one set of Q/K/V projections and one resulting pattern of attention weights per sequence. Any relationship the model needs to capture — subject-verb agreement, pronoun coreference, local word order, long-range topic relevance — all has to be expressed through that single set of learned weights and the single attention pattern it produces. WHY THIS IS ANALOGOUS TO THE CONSTRAINT llm1-4 ALREADY FLAGGED ------------------------------ Per this chapter, this is "exactly the kind llm1-4's own finding-box already flagged for forcing Query and Key into one shared vector" — there, collapsing two genuinely different roles (searching vs. offering) into one vector was shown to be an unnecessary constraint. Here, the same underlying problem reappears one level up: collapsing many genuinely different kinds of relationships into one single attention pattern forces the model to find one compromise representation good enough for all of them at once, rather than letting each relationship type be represented well on its own terms. CONCRETE EXAMPLE OF WHY THIS MATTERS ------------------------------ Per this chapter, one head "might learn to track subject-verb agreement, another might learn coreference... another might attend mostly to nearby, local context." These are structurally different kinds of attention patterns — subject-verb agreement often needs to look across a long span, local context needs to look at immediate neighbors, coreference needs to match specific noun-pronoun pairs regardless of distance. A single attention computation, with a single learned weighting, would have to find some blended compromise pattern serving all three needs simultaneously, rather than a pattern well- suited to any one of them individually. HOW MULTI-HEAD ATTENTION ADDRESSES THIS ------------------------------ Per this chapter's own code, multiple heads each get "their own independent set of learned W_Q/W_K/W_V matrices," computing attention "independently and in parallel." This means each head is free to specialize in whichever kind of relationship its own training signal pushes it toward, without needing to compromise with what any other head is doing. Concatenating all the heads' own outputs and projecting them back down through W_O then combines these separately-specialized views into one richer final representation per token — richer than any single, forced-to-compromise attention pattern could produce alone. WHY THIS WORKS AS AN ANSWER ------------------------------ It identifies the specific representational compromise a single attention computation is forced into, connects it explicitly to the same kind of constraint llm1-4 already named for Query/Key, gives a concrete example of genuinely different relationship types that would otherwise have to share one pattern, and explains how independent, parallel heads let each specialize instead.