Exercise 3: Frozen vs. Fine-Tuned, and Why It's the Same Shape as llm1's Pretrain-Then-Finetune Paradigm — Possible Solution ==================================================================== WHEN FREEZING IS THE RIGHT CHOICE ------------------------------ Per this chapter's own compare-table, freezing means the pretrained vectors are loaded and never updated during training — the right call specifically when the task-specific dataset is small. With little data to train on, allowing the embeddings to keep changing risks distorting them based on too few examples, the exact same small-dataset noise problem Exercise 1 already identified for training word2vec entirely from scratch. Freezing protects the pretrained vectors' own broad, well-supported structure from being degraded by a dataset too small to safely refine it further. WHEN FINE-TUNING IS THE RIGHT CHOICE ------------------------------ Fine-tuning means the pretrained vectors are loaded as a starting point, then continue updating during training. This is the right call when there is enough task-specific data to adapt the vectors to this particular domain — enough examples that the updates reflect genuine, supported patterns in this specific task rather than noise, allowing the embeddings to specialize (e.g. picking up domain-specific nuances) without losing the broad semantic structure they started from. WHY THIS IS A GENUINE TRADE-OFF, NOT A DEFAULT CHOICE ------------------------------ Neither option is universally correct — the right choice depends specifically on how much task-specific data is available. Too little data with fine-tuning enabled risks the same kind of unreliable, noise-driven vectors Exercise 1 diagnosed for from-scratch training; freezing when there's actually plenty of task-specific data leaves useful specialization on the table by never letting the model adapt at all. WHY THIS IS THE SAME SHAPE AS llm1's PRETRAIN-THEN-FINETUNE PARADIGM ------------------------------ Per this chapter's own finding-box, "a model trained once on a massive, general task, then either used as-is or further adapted on a smaller, specific one" describes both this chapter's frozen/fine-tuned choice and llm1's own pretrain-then-finetune paradigm identically. In both cases: (1) something is trained first on a large, general task (GloVe on a massive corpus; an LLM on a massive pretraining corpus), (2) a choice is then made about whether to use that result as-is (frozen embeddings; a frozen/base pretrained model) or continue adapting it on a smaller, specific dataset (fine-tuned embeddings; a fine-tuned model). WHY ONLY THE SCALE DIFFERS, NOT THE LOGIC ------------------------------ Per this chapter, "the only real difference is scale: here it's a lookup table of word vectors; there it's an entire deep network with billions of parameters." The decision logic — don't relearn from nothing what's already been learned well elsewhere, and decide whether to further specialize based on how much task-specific data is available — is identical in both cases; only the size and complexity of what's being reused differs. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains the specific data-availability condition that makes freezing versus fine-tuning the right choice in each case, and shows precisely how this chapter's own decision structure maps onto llm1's pretrain-then-finetune paradigm feature for feature, differing only in scale rather than in underlying logic.