Exercise 2: Why SFT Alone Can't Capture Preference, and What the Reward Model Adds — Possible Solution ==================================================================== WHAT SFT ACTUALLY TEACHES THE MODEL ------------------------------ Per this chapter, SFT trains the model on curated (instruction, ideal response) pairs using the same next-token-prediction objective from llm1-7. This means the model is learning to imitate the specific example responses it was shown — for any given prompt in the training set, it learns to produce something close to the one ideal response it was given for that exact prompt. WHY IMITATING EXAMPLES ISN'T THE SAME AS LEARNING PREFERENCE ------------------------------ Per this chapter, SFT "only teaches imitation of the specific examples shown — it can't, by itself, capture the subtler, harder-to-write-down human preferences about what makes one otherwise-reasonable response genuinely better than another." Two different responses to the same prompt might both be reasonable, correct, and well-written, yet humans might still consistently prefer one over the other for reasons that are hard to fully specify in advance — tone, structure, level of detail, how directly it addresses the actual question. Writing enough example responses to implicitly teach every such preference through imitation alone would require an impractically large and exhaustively comprehensive set of examples. WHAT THE REWARD MODEL SPECIFICALLY ADDS ------------------------------ Per this chapter's own code, the reward model is trained on a genuinely different kind of signal: rather than "here is the one ideal response," it learns from "here are several different responses to the same prompt, ranked by human preference from best to worst." This ranking-based signal directly captures comparative preference — not "what does a good response look like in isolation," but "given several plausible options, which do humans actually prefer, and by how much." WHY THIS IS A GENUINELY DIFFERENT KIND OF LEARNING SIGNAL ------------------------------ SFT's own training signal is a single fixed target per prompt (the one ideal response written for it). The reward model's own training signal is comparative and relative — it learns a scoring function that can evaluate and rank many different possible responses to the same prompt, capturing preference as a matter of degree rather than a single correct-or-incorrect target. This scoring function is exactly what Step 3 (RLHF) then needs, since RLHF requires a way to score arbitrary, freshly-generated responses that were never part of any fixed training example. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains precisely what SFT does and does not teach the model, using this chapter's own description, and shows how the reward model's own ranking-based training signal captures comparative human preference in a way a fixed set of imitation examples structurally cannot.