Exercise 1: Why Common Fragments Become Tokens While Rare Words Stay Fragmented — Possible Solution ==================================================================== WHAT THE BPE TRAINING LOOP ACTUALLY OPTIMIZES FOR ------------------------------ Per this chapter's own training loop, at every step BPE counts every adjacent pair of symbols across the entire training corpus and merges specifically "the single most frequent pair." This is a purely frequency-driven process — nothing in the algorithm asks whether a merge produces something meaningful, only whether it produces something common. WHY FRAGMENTS LIKE "ing" OR "tion" WIN EARLY MERGES ------------------------------ Per this chapter, "common fragments like 'tion', 'ing', or 'un' earn their own token early, since they occur constantly across a large corpus." Across millions of words of training text, the character sequence "ing" appears at the end of an enormous number of words (running, jumping, thinking, and so on), making the pair-count for "i"+"n" and then "in"+"g" extremely high very early in training. Since the algorithm always merges whatever pair currently has the highest count, these extremely common fragments get merged into single tokens long before rarer patterns do. WHY RARE WORDS DON'T GET THIS SAME TREATMENT ------------------------------ Per this chapter's own worked example, "rare words never get merged into single tokens at all — they stay broken into smaller, more common pieces." A word that appears only a handful of times in the entire training corpus never generates enough pair-count evidence to compete with fragments that appear millions of times. The algorithm has no special mechanism for "important but rare" words — it only ever follows frequency, so a rare word's own internal character pairs simply never rise high enough in the count to earn a merge. WHY THIS PRODUCES A GENUINELY DIFFERENT KIND OF VOCABULARY THAN nlp1-1's OWN ------------------------------ Because merges are chosen purely by corpus-wide frequency rather than by any notion of what a word means, the resulting vocabulary reflects statistical regularities in how characters co-occur, not the linguistically-motivated word/stopword/lemma distinctions nlp1-1's own pipeline was built around. A frequent but semantically empty fragment like "tion" earns a token exactly as readily as a frequent, meaningful whole word would — the algorithm makes no distinction between the two. WHY THIS WORKS AS AN ANSWER ------------------------------ It traces through this chapter's own training loop to show precisely why the algorithm is purely frequency-driven, explains why common fragments accumulate enough pair-count evidence to be merged early while rare words never do, and connects this to why the resulting vocabulary reflects statistical frequency rather than the linguistic structure nlp1-1's own pipeline was built around.