Exercise 3: Sinusoidal vs. Learned Positional Encoding, and Why Only One Has a Hard Maximum — Possible Solution ==================================================================== WHAT SINUSOIDAL POSITIONAL ENCODING ACTUALLY IS ------------------------------ Per this chapter, sinusoidal encoding computes each position's own vector directly from a fixed mathematical formula — sine and cosine waves at different frequencies, applied to the position number itself. Nothing about this formula is learned from data; it's the same calculation regardless of what corpus the model was trained on. WHAT LEARNED POSITIONAL EMBEDDINGS ARE INSTEAD ------------------------------ Per this chapter, a learned positional embedding table stores one distinct, trainable vector per position index, up to some fixed maximum the table was built with — position 0 has its own vector, position 1 has its own separate vector, and so on, each one adjusted during training exactly like the token embeddings themselves. THE TRADE-OFF, PER THIS CHAPTER'S OWN COMPARE-TABLE ------------------------------ Sinusoidal encoding requires no training and has "well-behaved relative-offset structure," per this chapter's own description of the original Transformer paper's finding that a fixed relative offset corresponds to a simple transformation of the encoding. Its downside is that it isn't adapted to any particular model's own training data — it's the same formula no matter what. Learned embeddings can adapt to whatever positional patterns are actually useful for this specific model and dataset, at the cost of that adaptation only extending as far as training actually went. WHY A LEARNED TABLE HAS A HARD MAXIMUM SEQUENCE LENGTH ------------------------------ Per this chapter's own warn-box, "a learned positional embedding table has a fixed size — it simply has no entry for position 8,193 if it was only ever trained up to position 8,192." A lookup table, by its very nature, can only contain entries for the specific indices it was built and trained with. There is no row in the table for a position that was never included during training, and no mechanism to invent one on demand — the table either has an entry for a given position or it doesn't, with nothing in between. WHY A SINUSOIDAL FORMULA DOES NOT HAVE THIS SAME LIMITATION, IN PRINCIPLE ------------------------------ Because sinusoidal encoding is computed directly from a formula rather than looked up from a fixed-size table, it can, in principle, be evaluated for any position number at all — including positions never seen during training — simply by plugging that position into the same sine/cosine formula used for every other position. The formula doesn't run out of entries the way a table can, since it was never a table to begin with. WHY THIS WORKS AS AN ANSWER ------------------------------ It defines each approach precisely using this chapter's own description, restates the genuine trade-off between adaptability and fixed formulas, and explains specifically why a lookup-table-based approach has a hard structural ceiling that a directly-computed formula does not, tying this forward to the real context-window limitation llm1-10 covers in full.