Exercise 3: Why Self-Supervision Is What Makes Trillion-Token Training Feasible — Possible Solution ==================================================================== WHAT nlp1-2/nlp1-3's OWN SMALL DATASETS LOOKED LIKE ------------------------------ Per this chapter, nlp1-2 and nlp1-3 worked with "small, manually- inspectable datasets" — a handful of example sentences, small enough that every example could realistically be looked at, checked, and understood by a person working with the course. Any labels those datasets needed (like the spam/not-spam distinction used elsewhere in the NLP course) were feasible to assign or verify by hand precisely because there were so few examples to begin with. WHY HAND-LABELING DOESN'T SCALE TO TRILLIONS OF TOKENS ------------------------------ Per this chapter, "hand-labeling even a fraction of a trillion tokens is not remotely feasible." If each token, or even each sentence, needed a person to read it and assign some kind of label, the sheer volume involved — hundreds of billions to trillions of tokens — would require an amount of human labor completely disproportionate to any realistic project timeline or budget. This isn't a matter of degree compared to nlp1-2/nlp1-3's own datasets; it's a difference of many orders of magnitude that makes the same manual approach categorically impossible at this scale. WHAT SELF-SUPERVISION MEANS FOR THE "LABEL" IN THIS CASE ------------------------------ Per this chapter, "the 'label' for every position is simply the next token already present in the raw text." Unlike nlp1-2/nlp1-3's own datasets, which needed an external, separately-assigned label (spam or not spam) that the text itself didn't automatically contain, pretraining's own prediction target is just whatever token happens to come next in the existing text — information that was already present in the raw source, requiring no separate annotation step at all. WHY THIS SPECIFICALLY REMOVES THE FEASIBILITY BOTTLENECK ------------------------------ Because the "label" is derived automatically from the text's own existing structure rather than assigned by a person, any raw text at all — scraped from the web, digitized from books, or pulled from code repositories — becomes usable training data the moment it's collected, with zero manual annotation step in between. The bottleneck that made trillion-token hand-labeling infeasible (the need for a human to look at and label each example) simply doesn't exist for this kind of training signal, since the corpus effectively labels itself. WHY THIS WORKS AS AN ANSWER ------------------------------ It contrasts the scale and labeling requirements of nlp1-2/nlp1-3's own small datasets against real pretraining corpora, explains precisely why hand-labeling becomes infeasible at that scale, and shows how self- supervision's own label-from-the-text-itself property removes exactly the bottleneck that would otherwise make training at this scale impossible.