Exercise 2: Why Position 4 Can Attend Backward Only, and Why This Is Essential for Generation — Possible Solution
====================================================================

WHAT llm1-6's OWN CAUSAL MASKING MECHANISM DOES
------------------------------
Per llm1-6, causal masking sets the attention score for any "future"
position to negative infinity before softmax, guaranteeing that
position receives exactly zero attention weight. Applied to this
chapter's own five-token sequence ("The", "capital", "of", "France",
"is"), the token at position 4 ("is") has its attention scores against
positions 0 through 4 (itself and everything before it) computed
normally, but any score against a position beyond 4 would be masked to
negative infinity — except in this specific sequence, there simply is
no position 5 or later yet, since generation hasn't produced anything
past "is" at this point.

WHY THIS MATTERS EVEN WHEN THERE'S NOTHING "FUTURE" YET TO MASK
------------------------------
The masking rule is applied uniformly regardless of whether a genuine
future token happens to exist at a given moment — it's a structural
property of every attention computation in the model, not something
switched on only when relevant. This consistency is exactly what
matters: the same masking rule that would block "is" from seeing a
hypothetical token 5 during training is the rule guaranteeing "is" was
never trained to expect information from a position that, at real
generation time, genuinely doesn't exist yet.

WHY THIS RESTRICTION IS ESSENTIAL FOR STEP 5's OWN GENERATION LOOP TO MAKE SENSE
------------------------------
Per this chapter's own Step 5, generation works by predicting one token,
appending it, and repeating the entire process to predict the next one.
At the moment the model predicts what comes after "is," positions 5, 6,
and beyond genuinely do not exist — there is nothing there to attend
to. If the model had been trained without causal masking (as in BERT's
own bidirectional attention, per llm1-6/llm1-7), it would have learned
to rely on future context during training that simply cannot be
supplied at real generation time, producing exactly the training/
generation mismatch llm1-7's own compare-table already identified as
disqualifying masked LM from coherent generation.

WHY CAUSAL MASKING IS WHAT MAKES THE LOOP CONSISTENT, STEP BY STEP
------------------------------
Because every position, at every layer, was trained under the
identical restriction it faces during real generation (attend only to
itself and earlier positions), each new token predicted in the Step 5
loop is generated under exactly the same conditions the model was
trained to expect. Without this restriction, there would be no
guarantee that a model's own training-time behavior transfers correctly
to the genuinely sequential, one-token-at-a-time reality of actual
generation.

WHY THIS WORKS AS AN ANSWER
------------------------------
It explains precisely what causal masking enforces at position 4 in
this specific example, connects this to why the restriction holds
uniformly regardless of whether a future token currently exists, and
shows why this consistency between training-time restriction and
generation-time reality is exactly what makes Step 5's own repeated
prediction loop coherent rather than broken.