Exercise 2: What "Variance" Means for a Model, and Why a Forest Has Less of It — Possible Solution ==================================================================== ds1-6's OWN DEFINITION OF VARIANCE, APPLIED HERE ------------------------------ ds1-6 defined variance as a measure of spread — how much values differ from an average, computed from the average of squared distances. This chapter applies that same underlying idea to a model rather than to a single dataset's own values, per this chapter's own restated definition: "how much would this model's own predictions change if trained on a different random sample from the same population?" Instead of asking how spread out a set of numbers is, this version asks how spread out the RESULTING MODELS would be if you repeated the entire training process on different random samples drawn from the same underlying population. WHY A SINGLE UNCONSTRAINED TREE HAS HIGH VARIANCE ------------------------------ Per this chapter, "a single unconstrained tree (ml1-7) has high variance — a different training sample would grow a noticeably different tree." Per ml1-7, an unconstrained tree can grow until its own leaves memorize individual training rows — meaning its exact structure (which features get split on, at which thresholds, how deep each branch goes) is heavily shaped by the specific rows it happened to be trained on. Train that same tree-building process on a different random sample of employees or cars, and a meaningfully different set of splits and thresholds would likely emerge, since a different set of specific rows would be available to memorize. WHY A RANDOM FOREST HAS LOWER VARIANCE ------------------------------ Per this chapter, "a random forest has lower variance — averaging many trees stabilizes the result regardless of which specific sample any one tree happened to see." Per ml1-7's own reasoning, each individual tree inside a forest still overfits to its own particular random subset, but combining many such trees' own predictions together (via majority vote or averaging) means the overall forest's own final output is far less sensitive to any ONE tree's own idiosyncratic training sample. If the entire forest were retrained on a different overall random sample from the same population, most of the many individual trees would still independently pick up on the same genuine underlying patterns (per ml1-7's own signal-vs-noise reasoning), so the forest's own combined predictions would end up much more similar across the two training runs than a single tree's own predictions would. WHY THIS COUNTS AS THE SAME UNDERLYING VARIANCE CONCEPT ------------------------------ In both cases (ds1-6's own numeric spread and this chapter's own model-to-model spread), variance measures how much something differs across repeated instances — repeated draws of a number in ds1-6's own case, repeated training runs on different samples in this chapter's own case. The forest's own predictions vary less across those repeated training runs than a single tree's own predictions do, which is exactly what "lower variance" means in this extended, model-level sense of the term. WHY THIS WORKS AS AN ANSWER ------------------------------ It restates the chapter's own extension of ds1-6's variance concept to models, applies it concretely to explain why a single tree's own structure is highly sensitive to its specific training sample, and explains why averaging many differently-trained trees in a forest produces a final result that stays comparatively stable across different training samples.