Exercise 2: How Positional Encoding Restores the Order Information Self-Attention Lacks — Possible Solution
====================================================================

WHAT nlp1-8 PROVED WAS MISSING
------------------------------
Per nlp1-8's own closing section, self-attention computes pairwise
relevance scores between tokens with no reference to where each token
sits in the sequence — swapping two tokens produces the identical set
of pairwise relationships. This chapter's own recap restates it plainly:
"nothing about the raw token embeddings, on their own, encodes where in
the sequence a token appears."

WHAT POSITION-DEPENDENT INFORMATION THIS CHAPTER ADDS
------------------------------
Per this chapter's own combining step, `input_vector = token_embedding +
position_embedding`. Every position in the sequence has its own
distinct positional-encoding vector (via the sinusoidal formula or a
learned table), so adding it to a token's own embedding shifts that
token's final input vector in a way that depends specifically on which
position it occupies.

WORKING THROUGH "dog bites man" vs. "man bites dog"
------------------------------
Per this chapter's own revisit, in "dog bites man," the word "dog"
occupies position 0, so its input vector is `embedding("dog") +
position_embedding(0)`. In "man bites dog," "dog" instead occupies
position 2, so its input vector is `embedding("dog") +
position_embedding(2)`. Since `position_embedding(0)` and
`position_embedding(2)` are different vectors, the two occurrences of
"dog" — despite sharing the identical token embedding — arrive at the
attention layer as genuinely different input vectors.

WHY THIS IS EXACTLY THE INFORMATION nlp1-8 SHOWED WAS MISSING
------------------------------
Because self-attention itself has no built-in notion of position (per
nlp1-8), the only way for position to matter to its computations is for
position to already be baked into the vectors attention receives as
input. By making a word's own final vector partly a function of where
it sits in the sequence, positional encoding supplies exactly the
missing ingredient — not by changing how attention itself computes
relevance, but by ensuring the values attention operates on already
differ based on position before attention ever runs.

WHERE, EXACTLY, THIS HAPPENS IN THE PIPELINE
------------------------------
Per this chapter's own code, the addition of token embedding and
positional embedding happens once, at the very start of the pipeline,
producing the vector that "actually enters layer 1." Every attention
computation in every subsequent layer then operates on vectors that
already carry this positional information baked in from the start,
rather than attention needing any special position-aware mechanism of
its own.

WHY THIS WORKS AS AN ANSWER
------------------------------
It connects nlp1-8's own diagnosis of exactly what information
self-attention lacks to this chapter's own mechanism for supplying it,
works through the specific "dog" example step by step to show the
resulting vectors genuinely differ, and identifies precisely where in
the pipeline this fix is applied.