Exercise 1: Why Small Datasets Produce Noisy Word2Vec Vectors, and What Pretrained Embeddings Fix — Possible Solution ==================================================================== WHAT WORD2VEC ACTUALLY NEEDS TO LEARN A GOOD VECTOR ------------------------------ Per nlp1-5, word2vec learns a word's own vector by repeatedly observing which words tend to appear near it across many training examples — the skip-gram/CBOW prediction task only gradually pushes semantically related words (like "excellent" and "great") toward similar vectors as the model sees both of them used in similar contexts, over and over, across many examples. WHY THE SPAM AND TOY DATASETS ARE TOO SMALL FOR THIS ------------------------------ The spam dataset from nlp1-2 and the toy sentences carried through nlp1-6/nlp1-7 contain only a handful of example sentences each — nowhere near the volume of repeated, varied context needed for word2vec's own prediction task to reliably discover real semantic relationships. A word appearing only once or twice in the entire dataset gives the training process almost nothing to learn from; its resulting vector would be shaped by whatever few contexts it happened to appear in, rather than by any genuine, broadly-supported pattern. WHY THIS PRODUCES NOISY, UNRELIABLE VECTORS SPECIFICALLY ------------------------------ Per this chapter, "training on too little text produces noisy, unreliable vectors no matter how correct the algorithm is." The algorithm itself isn't the problem — skip-gram/CBOW works exactly as designed. The problem is that with too few examples, the vectors it produces are essentially fitted to noise and coincidence in the small dataset rather than to real, stable semantic relationships that would only become visible across a much larger, more varied corpus. WHAT PRETRAINED EMBEDDINGS LIKE GLOVE SPECIFICALLY FIX ------------------------------ Per this chapter, GloVe is trained on massive corpora — Wikipedia, Common Crawl, billions of words. This gives its own co-occurrence statistics enormously more support than any small task-specific dataset could ever provide on its own, meaning relationships like "excellent" being near "great" are backed by genuinely broad, repeated evidence rather than a handful of coincidental sentences. Loading GloVe's own pretrained vectors means starting from representations that already reflect real semantic structure, sidestepping the small-dataset problem entirely rather than trying to fix it by training harder on the same limited data. WHY THIS WORKS AS AN ANSWER ------------------------------ It connects word2vec's own training mechanism from nlp1-5 to why it specifically requires large amounts of repeated context, shows why the small datasets used earlier in this course fall far short of that requirement, and explains precisely how GloVe's own massive-corpus training sidesteps the problem rather than merely mitigating it.