Exercise 3: Why an Unshuffled Split Is ds1-6's Own Sampling Bias, One Layer Up — Possible Solution ==================================================================== ds1-6's OWN DEFINITION OF SAMPLING BIAS ------------------------------ Per ds1-6, sampling bias is "a flawed sampling method producing a misleading conclusion even when every individual calculation performed on it is done correctly," illustrated with a weekday-only sales sample that could never reveal a weekend-specific dip no matter how carefully the resulting numbers were analyzed afterward. WHAT AN UNSHUFFLED YEAR-SORTED SPLIT ACTUALLY DOES ------------------------------ Per this chapter, "if ds1-9's own used-car listings were sorted by year and split straight down the middle with no shuffling, the test set could end up containing only the newest cars while training saw only older ones." The SPLITTING PROCESS ITSELF — deciding which rows go into training and which go into testing — is a sampling method, exactly the same category of thing ds1-6's own weekday-only selection was. Sorting by year and splitting down the middle systematically assigns rows to each set based on age rather than assigning them representatively, the same structural flaw as selecting only weekday transactions. WHY THIS COUNTS AS THE SAME CATEGORY OF MISTAKE, NOT JUST A SIMILAR ONE ------------------------------ Per this chapter's own tip-box, "an unshuffled train/test split is exactly that same failure, one layer up: the split itself is the sampling method, and getting it wrong produces a misleading performance number no matter how correctly every later calculation is done." This directly echoes ds1-6's own "even when every individual calculation... is done correctly" framing — a model trained only on older cars and tested only on newer ones (or vice versa) could have its accuracy, error metrics, and every other later computation performed with perfect mathematical correctness, and the resulting test score would still be meaningless as a measure of how the model performs on a genuinely representative mix of cars, because the test set itself was never representative to begin with. WHY "ONE LAYER UP" IS THE RIGHT WAY TO DESCRIBE IT ------------------------------ ds1-6's own example was about sampling FROM A POPULATION to get a dataset in the first place. This chapter's own example is about sampling FROM AN ALREADY-COLLECTED DATASET to create a train/test split — a second, later sampling step, applied to data that's already been gathered, rather than the original data-collection step itself. The underlying failure mode (a systematic, non-representative selection producing a misleading result regardless of downstream calculation accuracy) is identical; only the stage of the pipeline at which it occurs has moved one step later. WHY THIS WORKS AS AN ANSWER ------------------------------ It restates ds1-6's own definition of sampling bias precisely, applies it directly to the mechanics of an unshuffled, sorted train/test split, and explains why the phrase "one layer up" accurately describes the relationship — the identical failure mode, recurring at a later stage of the same overall data pipeline.