Exercise 2: How Causal Masking Works Mechanically, and Why -inf Rather Than Zero — Possible Solution ==================================================================== WHAT THIS CHAPTER'S OWN CODE ACTUALLY DOES, STEP BY STEP ------------------------------ `scores = (Q @ K.T) / sqrt(d_k)` computes the same raw attention scores as llm1-4's own unmasked version. `scores = scores.masked_fill (future_positions, -inf)` then overwrites every score corresponding to a "future" token (any position later in the sequence than the token doing the attending) with negative infinity, leaving scores for the current and earlier positions untouched. `weights = softmax(scores, axis=-1)` then converts the modified scores into attention weights, and `output = weights @ V` blends Value vectors using those weights, exactly as in llm1-4's own unmasked mechanism. WHY -inf SPECIFICALLY, AND NOT SIMPLY ZERO ------------------------------ Softmax works by exponentiating each input score and then dividing by the sum of all the exponentials, so that the outputs form a proper probability distribution summing to 1. If a future position's raw score were set to 0 instead of -inf, softmax would still assign it a real, positive exponential value (e^0 = 1), meaning that position would still receive a non-zero share of the attention weight — exactly the "peeking" at future tokens causal masking is meant to prevent. Setting the score to negative infinity instead means its exponential is exactly 0 (e^-inf = 0), so after normalization, that position genuinely receives 0 attention weight — not a small amount, exactly none. WHY EXACTLY ZERO WEIGHT MATTERS HERE, NOT JUST A SMALL WEIGHT ------------------------------ Per this chapter, "a token generating its own prediction can never 'peek' at tokens that haven't been generated yet." If future tokens received even a small non-zero weight, information from those tokens would still leak into the current token's own output vector, however slightly — which would make the model's own training and generation processes inconsistent with each other, since during real generation those future tokens genuinely don't exist yet to be attended to at all. Using -inf guarantees the leak is not merely reduced but eliminated. WHY THIS IS THE MECHANISM THAT MAKES GENERATION COHERENT ------------------------------ Per this chapter, this single change "is what makes autoregressive generation coherent: predicting each next token uses only what has genuinely already been produced." Because training with causal masking forces the model to predict each token using strictly zero information from later positions, the exact same constraint holds true at real generation time, when later tokens genuinely don't exist yet — training and generation stay consistent with each other specifically because the masking guarantees zero leakage, not merely low leakage. WHY THIS WORKS AS AN ANSWER ------------------------------ It traces through this chapter's own masked attention code line by line, explains precisely why softmax's own exponentiation step requires -inf rather than 0 to produce a genuinely zero attention weight, and connects this to why exact zero (not merely small) future-position weight is what keeps training and generation behavior consistent.