Exercise 2: Why "viagra" and "now" Score So Differently Despite Both Being Frequent — Possible Solution ==================================================================== WHAT THE TWO WORDS SHARE, PER THIS CHAPTER'S OWN TABLE ------------------------------ Per this chapter's own comparison table, both "now" and "viagra" are marked "Yes" (or "Yes, here") under "Frequent in this doc?" — within the one specific document being scored, both words genuinely occur with meaningful frequency. If TF-IDF depended only on TF, these two words would score similarly. WHERE THE TWO WORDS DIVERGE, PER THE SAME TABLE ------------------------------ The table's own second column, "Common across corpus?", is where the two words split: "now" is marked "Yes" (common across the corpus — it appears in most messages, spam or not), while "viagra" is marked "No" (rare across the corpus — it appears only in a distinctive subset of messages). WHY THIS DIFFERENCE DRIVES THE TF-IDF SCORE, PER THE IDF FORMULA ------------------------------ Per this chapter, IDF(word) = log(total documents / documents containing word). "now" appearing in most documents means the ratio (total documents / documents containing "now") is close to 1, so its own IDF is close to log(1) = 0 (per Exercise 1's own reasoning) — a low IDF. "viagra" appearing in only a small number of documents means the ratio (total documents / documents containing "viagra") is large, so its own IDF is a meaningfully larger, positive number — a high IDF. WHY BOTH WORDS' OWN TF DOESN'T MATTER MUCH HERE ------------------------------ Per this chapter, "TF-IDF = TF × IDF." Since both words share a similarly high TF within this specific document, TF isn't what distinguishes their final scores — IDF is doing essentially all the differentiating work. "now"'s own low IDF pulls its overall TF-IDF score down toward zero regardless of its high TF, per this chapter's own table entry marking its resulting score "Low." "viagra"'s own high IDF, combined with its high TF, produces a genuinely high overall TF-IDF score, per the table's own "High" entry. WHY THIS MATCHES THE CHAPTER'S OWN EARLIER MOTIVATING EXAMPLE ------------------------------ This is precisely the concrete version of the chapter's own opening claim: "'now' might appear often in a spam message — and just as often in an ordinary one. Its raw count is high, but it does little to actually distinguish spam from not-spam. A genuinely rare word like 'viagra'... is far more informative." TF-IDF's own scoring mechanism — multiplying frequency-within-document by rarity-across-corpus — is specifically designed to produce exactly this outcome: rewarding words that are both common here and rare elsewhere, while suppressing words that are common everywhere regardless of how often they appear in any one document. WHY THIS WORKS AS AN ANSWER ------------------------------ It identifies precisely which column of the chapter's own table the two words share (frequency within the document) and which column they diverge on (rarity across the corpus), and traces that divergence through the IDF formula to explain why it — not TF — is what produces their dramatically different final TF-IDF scores.