Exercise 1: Why "High Error on Both" Rules Out Overfitting — Possible Solution ==================================================================== WHAT EACH FAILURE MODE PRODUCES, PER THIS CHAPTER'S OWN TABLE ------------------------------ Per this chapter's own compare-table: overfitting shows "Low" training error and "High" test error; underfitting shows "High" error on both training AND test. WHY OVERFITTING SPECIFICALLY REQUIRES A GAP ------------------------------ Per this chapter, overfitting happens when a model is "too complex — fits noise/idiosyncrasies in the training data." A model that has successfully fit even the training data's own noise will, by definition, perform very well ON that same training data — that's precisely what "fitting the noise" means: matching the training examples closely, including their quirks. This is exactly why overfitting necessarily produces LOW training error — the defining behavior of overfitting is doing unusually well on the data it directly saw. The test-set error stays high because those training-specific quirks don't generalize to new, unseen data. WHY HIGH TRAINING ERROR IS INCOMPATIBLE WITH OVERFITTING ------------------------------ If a model has HIGH error even on its own training data, it has, by definition, failed to closely fit that data at all — including failing to fit whatever noise or idiosyncrasies exist in it. Since overfitting specifically requires successfully fitting the training data closely (low training error) as its defining symptom, a model performing poorly even on training data cannot be overfitting by definition — there's no close fit to speak of, noise-driven or otherwise. WHY THIS LEAVES UNDERFITTING AS THE ONLY OTHER OPTION ------------------------------ Per this chapter, underfitting occurs when "the model is too simple to capture the real pattern" — a model lacking the flexibility to model even the genuine, real relationships present in the training data, independent of any noise at all. This directly produces high error on BOTH datasets: the model was never capable of fitting the real, underlying pattern in the first place, so it performs poorly whether tested on data it trained on or genuinely new data — there was never anything for it to "successfully fit" on either set. WHY THE GAP ITSELF IS THE ACTUAL DIAGNOSTIC SIGNAL ------------------------------ Per this chapter, "the diagnostic signature is the gap." The single most informative piece of information isn't either error number in isolation, but the RELATIONSHIP between them: a large train-test gap signals overfitting (a model that fit training-specific quirks that don't generalize); no meaningful gap, with both numbers poor, signals underfitting (a model that never fit anything meaningful, training data included). This is precisely why "high error on both" mechanically rules out overfitting — overfitting's defining feature (a strong fit to training data specifically) is simply absent when training error itself is high. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains why overfitting's own definition mechanically requires low training error as a symptom, and why a model performing poorly on training data therefore cannot meet that definition — leaving underfitting, whose own definition produces uniformly poor performance on both datasets, as the only consistent diagnosis.