Exercise 1: Why N Tokens Yield N Training Signals, Computed in Parallel — Possible Solution ==================================================================== WHY EVERY POSITION IN THE SEQUENCE PRODUCES ITS OWN PREDICTION TARGET ------------------------------ Per this chapter's own worked example, the sequence "The capital of France is Paris" produces a separate prediction target at every position: "The" alone must predict "capital," "The capital" together must predict "of," and so on, up through "The capital of France is" predicting "Paris." Each of these is a genuinely separate training example — a different amount of preceding context, paired with a different correct next token — all drawn from the exact same six-token sequence. WHY THIS PRODUCES N SIGNALS FOR N TOKENS ------------------------------ A sequence of N tokens has N positions, and per this chapter's own causal language modeling objective, "the model's job at every single position is to predict the token that actually comes next." Since every one of the N positions (except trivially the very last, which has nothing after it to predict) generates its own independent prediction-versus-actual-next-token comparison, one sequence yields roughly N usable training signals rather than just one. WHY THIS CAN BE COMPUTED IN PARALLEL RATHER THAN SEQUENTIALLY ------------------------------ Per this chapter, this is possible "because llm1-6's own causal mask already prevents any position from seeing ahead, every position's prediction can be trained simultaneously, in parallel, rather than one at a time." The causal mask (llm1-6) guarantees that the prediction at position 3, say, mathematically cannot depend on anything from position 4 or later, regardless of the order those computations happen to run in. Because none of the per-position predictions depend on any other position's own output, there's no need to compute them one after another in sequence — all of them can be computed at once, in a single forward pass through the network. WHY THIS IS A GENUINE EFFICIENCY ADVANTAGE OVER nlp1-6's OWN LSTM TRAINING ------------------------------ Per this chapter, nlp1-6's own LSTM training was "strictly sequential," since nn1-8's own hidden-state mechanism means "each step had to wait for the one before it" — step 5's computation genuinely cannot begin until step 4's own hidden state has been produced. Causal self- attention has no equivalent dependency chain between positions during a single forward pass, since attention scores between any two positions can be computed independently of every other pair. This is precisely why a Transformer can extract N separate training signals from one sequence in roughly the same amount of computation an LSTM would need for far fewer. WHY THIS WORKS AS AN ANSWER ------------------------------ It walks through this chapter's own worked example to show exactly why each position produces its own distinct prediction target, connects this to why causal masking removes any cross-position dependency that would force sequential computation, and contrasts this directly with nlp1-6's own genuinely sequential LSTM training to explain the real efficiency gain.