Exercise 2: What a Validation Set Fixes That a Plain Train/Test Split Doesn't — Possible Solution ==================================================================== WHY A PLAIN TRAIN/TEST SPLIT BREAKS DOWN UNDER ITERATIVE TUNING ------------------------------ Per this chapter, "if a model is trained, checked against the test set, adjusted based on what went wrong, checked against the test set again, adjusted again — the test set is no longer serving its original purpose. Even without formally training on it, repeatedly using test-set performance to guide decisions lets information about the test set quietly leak into the model through every one of those adjustment cycles." WHY THIS COUNTS AS "LEAKING," EVEN WITHOUT FORMAL TRAINING ------------------------------ Each time a decision (adjust this setting, try a different approach) is made specifically BECAUSE it improved test-set performance, that decision has been shaped by information about the test set — even though the model's own internal weights were never directly fit to it. Over many rounds of this, the choices that survive are, collectively, the ones that happen to work well specifically on this one test set — which is functionally similar to fitting to it, just achieved through a sequence of human or automated decisions rather than through the model's own training algorithm directly. WHY THE FINAL TEST SCORE BECOMES DISHONEST AS A RESULT ------------------------------ Per this chapter, "the final 'test accuracy' ends up flattered by the same underlying problem this chapter opened with, just one step more indirect." The test set's entire purpose was to measure performance on data that had no influence on the model at all — but after many rounds of tuning guided by that same test set's own results, it no longer represents fully independent, unseen data. The reported score becomes optimistic in the same fundamental way a training-data score is optimistic, just reached through a more roundabout, iterative path. WHAT A VALIDATION SET SPECIFICALLY FIXES ------------------------------ Per this chapter's own table, the validation set exists to be "used for exactly this kind of iterative tuning, while the test set stays completely untouched until one single, final check." By giving the tuning process its own dedicated dataset — one that's allowed to be repeatedly checked and used to guide decisions — the test set is freed from ever being touched during development at all. When the test set is finally checked, exactly once, at the very end, none of the earlier tuning decisions could possibly have been shaped by it, so its score genuinely reflects performance on data the whole development process never saw. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains precisely why repeated test-set checking constitutes a real form of leakage even without formal training on that data, and explains how introducing a separate validation set — dedicated specifically to iterative tuning — restores the test set's own original purpose by removing any reason to touch it before the single final evaluation.