Exercise 2: Why Masked LM Can't Generate Coherently, But Causal LM Can — Possible Solution ==================================================================== WHAT BERT'S OWN MASKED LANGUAGE MODELING OBJECTIVE ACTUALLY TRAINS FOR ------------------------------ Per this chapter's own compare-table, masked LM trains the model using "both directions (bidirectional)" context — when predicting a masked token, the model is allowed to look at every other token in the sequence, including ones that come after the masked position. Per llm1-6, this is the same underlying idea as nlp1-7's own bidirectional LSTM: the token being predicted is surrounded on both sides by real, already-present context during training. WHY THIS CREATES A MISMATCH AT REAL GENERATION TIME ------------------------------ Per this chapter, masked LM "assumes the whole sequence already exists" — during training, every token other than the masked ones is genuinely present and available to attend to. But generating text left to right means producing one token at a time, with no later tokens existing yet at the moment each one is produced. A model trained to rely on both-directions context has no consistent way to operate when half of that context (everything after the current position) is simply absent, because it was never trained to make predictions under that condition. WHAT CAUSAL LANGUAGE MODELING TRAINS FOR INSTEAD ------------------------------ Per this chapter's own compare-table, causal LM restricts context to "only earlier tokens (causal)" during training — the exact same restriction llm1-6's own causal mask enforces during generation. Every prediction the model is ever trained to make already only uses information that would also be genuinely available at real generation time. WHY THIS MAKES CAUSAL LM CONSISTENT WITH GENERATION AND MASKED LM INCONSISTENT ------------------------------ Per this chapter's own compare-table, causal LM is explicitly marked "Yes — training and generation are consistent," while masked LM is marked "No — assumes the whole sequence already exists." The core issue is a mismatch between the conditions present during training and the conditions present during actual use: causal LM trains under exactly the same information constraints it will face at generation time, while masked LM trains under a strictly easier condition (full bidirectional context) that generation can never actually provide. WHY THIS EXPLAINS llm1-6's OWN ARCHITECTURE SPLIT ------------------------------ Per this chapter, this mismatch is "the concrete, mechanical reason llm1-6's own architecture split holds." BERT's own bidirectional design and masked LM objective are well-suited to understanding tasks, where the whole input is available at once and there's no generation step at all — but that same design structurally cannot be reused for coherent left-to-right generation, which is exactly why GPT's own causally-masked, causal-LM-trained architecture became the dominant choice for general-purpose generation instead. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains what each objective actually trains the model to rely on, identifies the specific mismatch between masked LM's own training condition and the reality of left-to-right generation, and connects this directly to why llm1-6's own architecture families split the way they did.