Exercise 1: CBOW vs. Skip-Gram, and Why Both Are Genuinely Self-Supervised — Possible Solution ==================================================================== THE TWO APPROACHES, PER THIS CHAPTER'S OWN TABLE ------------------------------ Per this chapter's own compare-table: CBOW "predicts the target word, from its surrounding context words," while skip-gram "predicts the surrounding context words, from the target word." The two are exact mirror images of each other — CBOW goes from context to target, skip- gram goes from target to context. WHAT THIS MEANS CONCRETELY ------------------------------ Given a sentence like "the cat sat on the mat," with "sat" as the target word and the surrounding words as context: CBOW's own training task is, given the context words ("the," "cat," "on," "the," "mat"), predict that the missing word in the middle is "sat." Skip-gram's own training task runs the opposite direction: given the single word "sat," predict that the surrounding context words are likely to include "the," "cat," "on," "the," "mat." WHY BOTH ARE GENUINELY SELF-SUPERVISED ------------------------------ Per this chapter, "the 'labels' here are just other words already present in the raw text — the task supervises itself." In both CBOW and skip-gram, the "correct answer" the network is trained to predict is never separately provided by a human annotator at all — it's simply another word that already existed, unlabeled, in the exact same sentence the input words came from. The raw text itself provides both the input and the target simultaneously, with the input/target roles merely assigned by which words happen to sit near each other in a sentence that was written for entirely ordinary reasons, with no NLP training purpose in mind. WHY THIS DIFFERS FROM ORDINARY SUPERVISED LEARNING ------------------------------ Per ml1-1's own distinction, supervised learning (like ml1-5's own logistic regression) requires a labeled dataset where each example's own correct answer was determined and recorded by some external process before training — a human deciding whether an employee actually left the company, for instance. CBOW and skip-gram require no equivalent external labeling step at all; the "label" for any given training example is generated automatically and mechanically just by looking at which words happen to sit near each other in already- existing text, which is exactly what makes both approaches self- supervised rather than supervised in the ordinary sense. WHY THIS MATTERS PRACTICALLY ------------------------------ Because no human labeling is required, word2vec can be trained on effectively unlimited quantities of raw, freely available text (news articles, books, web pages) rather than being limited to however much manually-labeled data happens to exist for a specific task — a real, practical advantage directly enabling the kind of large-scale training this chapter's own geometric payoff (semantically meaningful embedding space) actually depends on. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains the specific prediction direction each approach uses (context→target vs. target→context) with a worked example, and explains precisely why both qualify as self-supervised — the "label" being ordinary text that already existed, requiring no separate human annotation step at all.