Exercise 2: What Chinchilla Actually Found, and Why Kaplan's Lesson Was Misread — Possible Solution ==================================================================== WHAT KAPLAN'S ORIGINAL 2020 SCALING LAWS SUGGESTED ------------------------------ Per this chapter, "Kaplan et al.'s original 2020 scaling laws suggested that, for a fixed compute budget, model size should be scaled up much faster than dataset size." Interpreted straightforwardly, this pointed labs toward building the largest possible models, even if that meant training them on comparatively less data relative to their own size — per this chapter, GPT-3 is a real example of this era's approach. WHAT GOPHER REPRESENTED ------------------------------ Per this chapter's own compare-table, Gopher used 280 billion parameters, following the Kaplan-style approach of a large model with comparatively less data — and the chapter states plainly this left Gopher "undertrained relative to its own size." The model had far more capacity (parameters) than its training data was giving it the chance to actually use well. WHAT CHINCHILLA DID DIFFERENTLY, AND WHAT RESULTED ------------------------------ Per this chapter, Chinchilla used only 70 billion parameters — one-quarter of Gopher's size — but scaled "model size and dataset size ... roughly equally," using the identical compute budget Gopher had used. The result: Chinchilla, despite being dramatically smaller, outperformed the much larger Gopher. WHY THIS SHOWS "BIGGER MODEL, SAME COMPUTE" WAS THE WRONG LESSON ------------------------------ If Kaplan's own findings had been correctly interpreted as "for a fixed compute budget, there is a specific optimal split between model size and data size," Gopher's own designers might have chosen a smaller model trained on proportionally more data, the same choice Chinchilla's own designers made. Instead, the practical lesson many took from Kaplan's work was closer to "make the model as large as possible" — which, per this chapter's own finding, produced models that were "significantly undertrained relative to their own parameter count," leaving real performance on the table that a smaller, better-balanced model could capture using the identical compute budget. WHY THIS IS A GENUINE CORRECTION, NOT JUST A DIFFERENT OPINION ------------------------------ This isn't a matter of preference between two equally valid strategies — per this chapter, Chinchilla's own real, measured performance directly outperformed Gopher's, using the same amount of compute. That head-to-head result is what makes this a real, historically documented correction: the actual optimal ratio between model size and dataset size, for any given compute budget, is closer to Chinchilla's own balanced approach than to Gopher's own size-prioritizing one. WHY THIS WORKS AS AN ANSWER ------------------------------ It contrasts Gopher's and Chinchilla's own specific approaches and results using this chapter's own compare-table, and explains precisely why treating Kaplan's own findings as "make the model bigger" rather than "find the right size-to-data ratio for your compute budget" was the misreading Chinchilla's own results corrected.