Exercise 1: What Query, Key, and Value Represent, and Why Three Separate Matrices — Possible Solution ==================================================================== WHAT EACH VECTOR REPRESENTS, PER THIS CHAPTER ------------------------------ Query represents "what this token is currently trying to find out from the rest of the sequence" — it's the token's own search request. Key represents "what this token has to offer, in a form other tokens' Queries can be compared against" — it's the token's own advertisement of relevance, formatted specifically so it can be matched. Value represents "the actual content this token contributes once it's been attended to" — it's the substantive information that actually gets passed along once a token has been selected as relevant. WHY THIS THREE-WAY SPLIT MAKES SENSE CONCEPTUALLY ------------------------------ Searching for something, advertising what you have, and providing the actual substance once found are three genuinely different jobs. A librarian card-catalog analogy helps: the Query is like a search term typed into the catalog, the Key is like the catalog card describing what each book is about, and the Value is the book's own actual content, retrieved once the search term matches a card. All three relate to the same underlying token, but each captures a different aspect of its role in the computation. WHY A SINGLE SHARED VECTOR WOULD BE A REAL CONSTRAINT ------------------------------ Per this chapter's own finding-box, "if a token used the same vector for both roles, it would be forced to represent 'what I'm searching for' and 'what I actually contain' as the identical vector, which is a real, unnecessary constraint." Concretely: what a token is looking for (its Query) doesn't have to resemble what it itself offers (its Key) — a verb might be looking for a subject, while offering, as its own Key, information relevant to verbs. Collapsing Query and Key into the same vector would force the model to represent both of these very different things identically, losing the ability to distinguish "what I seek" from "what I am." WHY SEPARATE LEARNED MATRICES SOLVE THIS ------------------------------ Because Q, K, and V are each produced by their own distinct weight matrix (W_Q, W_K, W_V) applied to the same input vector, the model is free to learn three different projections of that same underlying token — one useful for searching, one useful for being matched against, and one useful for being the actual payload once matched. Training can shape each projection independently to best serve its own specific role, rather than being forced to compromise on a single shared representation. WHY THIS WORKS AS AN ANSWER ------------------------------ It restates what each of Query, Key, and Value represents using this chapter's own definitions, illustrates the distinction with a concrete analogy, and explains, using this chapter's own reasoning, exactly why collapsing all three into one shared vector would impose a real, unnecessary representational constraint that three separate learned matrices avoid.