Exercise 1: How Parameter Sharing Solves Both Problems at Once — Possible Solution ==================================================================== THE TWO PROBLEMS THIS CHAPTER IDENTIFIES ------------------------------ Per this chapter, a fully-connected layer applied to images faces two separate problems: an enormous parameter count ("roughly 150,000 weights per neuron" for a modest image), and no translation invariance ("shift an object slightly in the frame, and the network has to essentially relearn the same pattern all over again at the new position"). WHAT PARAMETER SHARING ACTUALLY MEANS, PER THIS CHAPTER ------------------------------ Per this chapter, "a kernel... slides across the image, computing a small, local weighted sum at each position. Crucially, the same set of weights is reused at every position." Rather than each spatial position in the image having its own independent, separately-learned set of weights (as a fully-connected layer would), one small kernel's own weights are applied identically at every single location the kernel slides to. WHY THIS DIRECTLY SOLVES THE PARAMETER-COUNT PROBLEM ------------------------------ Per this chapter, "parameter sharing collapses the parameter count from 'one weight per pixel' down to 'one small kernel, applied everywhere.'" Instead of needing a separate weight for every possible pixel position across the entire image (the fully-connected layer's own requirement), only the kernel's own small, fixed number of weights (for example, 9 weights for a 3×3 kernel) need to be learned at all — that same small set gets reused at every position rather than requiring its own independent copy per location. The total parameter count for one kernel is tiny and completely independent of the image's own size, unlike a fully-connected layer's own parameter count. WHY THIS SAME MECHANISM ALSO SOLVES TRANSLATION INVARIANCE ------------------------------ Per this chapter, "it also delivers translation invariance for free: a kernel trained to detect, say, a vertical edge detects that same edge wherever it appears in the image, because the identical weights are applied at every position rather than being learned separately for each one." Because the kernel's own weights are literally identical regardless of where in the image it's currently being applied, a pattern the kernel has learned to detect gets detected with EXACTLY the same sensitivity no matter where that pattern happens to appear in the image — there's no separate, position-specific set of weights that would need to independently "learn" the same pattern all over again at a new position, because there was never a position-specific set of weights to begin with. WHY BOTH FIXES COME FROM THE SAME SINGLE MECHANISM ------------------------------ Both problems trace back to the same root cause in a fully-connected layer: a separate, independent weight for every input position. Sharing one small set of weights across every position eliminates that root cause directly and simultaneously — fewer total weights to learn (fixing the parameter-count problem) and identical treatment of a pattern regardless of its position (fixing the translation-invariance problem) are two consequences of the exact same design choice, not two separate fixes that happen to be bundled together. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains precisely what parameter sharing means mechanically, and traces both of the chapter's own named problems back to the same root cause (independent per-position weights in a fully-connected layer), showing why eliminating that one root cause via kernel reuse solves both problems as a single, unified consequence rather than as two unrelated fixes.