Exercise 2: Why BPE's Byte-Level Fallback Fixes OOV "For Good," Not Just Less Often — Possible Solution ==================================================================== WHAT GLOVE'S OWN VOCABULARY STRUCTURE WAS ------------------------------ Per nlp1-9, GloVe's vocabulary is a fixed table built at training time — a finite list of whole words, each mapped to a precomputed vector. Anything not on that list has no entry at all; there is no smaller unit GloVe can fall back to, because "word" is the smallest unit GloVe's own vocabulary ever operates on. Adding more words to the table would shrink the problem, but never eliminate it — new jargon, typos, and novel words could always still exceed whatever fixed list was chosen. WHAT BPE'S OWN VOCABULARY STRUCTURE IS INSTEAD ------------------------------ Per this chapter, BPE's vocabulary is built bottom-up, starting from individual characters (or, in byte-level BPE, individual bytes) and merging upward based on frequency. Crucially, this means the vocabulary always still contains every one of those smallest units even after thousands of merges — whole words and common fragments get their own single tokens as a convenience, but the individual-character/byte level never disappears from the vocabulary. WHY THIS MAKES THE FIX ABSOLUTE RATHER THAN PARTIAL ------------------------------ Per this chapter's own finding-box, "literally any string, including gibberish nobody has ever typed before, can always be decomposed into some sequence of known vocabulary pieces." Because the smallest guaranteed unit (a byte) can represent any possible string in any script, there is no input that could ever exceed what the vocabulary can express — worst case, a totally novel string just gets broken all the way down to individual bytes, exactly as this chapter's own "zzyxplorb" example shows. GloVe's fix (a bigger word list) only ever narrows the set of failing inputs; BPE's fix (a byte-level fallback) eliminates the category of failing inputs entirely, since there is no string that cannot be expressed as some sequence of bytes. WHY GROWING GLOVE'S OWN VOCABULARY COULD NEVER ACHIEVE THE SAME GUARANTEE ------------------------------ No matter how large a fixed word list becomes, new words, deliberate misspellings, and invented terms will keep appearing, and each one remains a genuine gap unless anticipated in advance. BPE doesn't need to anticipate every future word — its fallback to bytes means new words are automatically representable using pieces the vocabulary already has, without ever needing to add a new entry for them. WHY THIS WORKS AS AN ANSWER ------------------------------ It contrasts GloVe's own fixed, whole-word-only vocabulary structure with BPE's own bottom-up, byte-level vocabulary structure, and explains specifically why only the latter can claim to solve out-of-vocabulary inputs completely rather than merely reducing how often they occur.