Exercise 1: What a Perfect Training Score Actually Tells You — Possible Solution ==================================================================== THE ANALOGY, PER THIS CHAPTER ------------------------------ Per this chapter, "imagine grading a student using the exact same practice questions they studied from, answer key included. A perfect score wouldn't tell you whether they actually understand the material — only whether they memorized those specific answers." WHY A PERFECT SCORE DOESN'T MEASURE UNDERSTANDING ------------------------------ A student who scores perfectly on the exact practice questions they studied could have achieved that score through either genuine comprehension of the underlying material OR pure rote memorization of those specific question-answer pairs, with no real understanding of the concepts behind them. The test itself — reusing the exact same questions — has no way to distinguish between these two very different possibilities, because both would produce an identical, perfect result on this particular measurement. WHY THIS MAPS DIRECTLY ONTO A MODEL EVALUATED ON TRAINING DATA ------------------------------ Per this chapter, "a machine learning model has exactly the same failure mode: shown enough examples, it can simply memorize the relationship between each specific input and its specific answer, rather than learning a pattern that generalizes to a new example it's never seen." A model tested on its own training data faces the identical ambiguity as the student: perfect performance is equally consistent with "the model learned a real, generalizable pattern" and "the model simply memorized these specific training examples" — the training-data score alone provides no way to tell which of those two happened. THE ONE THING A PERFECT TRAINING SCORE DOES TELL YOU ------------------------------ It confirms only that the model (or student) is CAPABLE of reproducing the correct answer for the specific examples it was directly exposed to during training/studying — nothing more. This is a genuinely weaker and less useful claim than "the model has learned the underlying pattern," since the first claim is trivially achievable by pure memorization even in the total absence of the second. WHY THIS IS WHY THE TEST SET MUST BE GENUINELY UNSEEN ------------------------------ Because a training-data score can't distinguish memorization from generalization, the only way to actually measure generalization — the thing that matters for a model's real usefulness on new, future data — is to check its performance on examples it was never shown during training at all, which is precisely why this chapter's own test set is defined as data "held back, completely untouched during training." WHY THIS WORKS AS AN ANSWER ------------------------------ It explains precisely why the analogy's own perfect score is ambiguous between memorization and real understanding, applies that same ambiguity directly to a model's training-data score, and identifies the one narrow, weaker claim (capability of reproducing seen answers) that a perfect training score does legitimately establish.