Exercise 1: TF-IDF's Two Components, and Why Combining Them Beats TF Alone — Possible Solution ==================================================================== TERM FREQUENCY (TF) ------------------------------ Per this chapter, "Term Frequency (TF): how many times does the search term appear in this document — a document mentioning 'wireless' five times is, all else equal, probably more relevant to a 'wireless' search than one mentioning it once." TF measures, within a SINGLE document, how often the search term shows up — more occurrences suggest the document is more strongly "about" that term. INVERSE DOCUMENT FREQUENCY (IDF) ------------------------------ Per this chapter, "Inverse Document Frequency (IDF): how rare is this term across the whole index — a term appearing in nearly every document tells you less about relevance than a term appearing in only a few. A document matching a rare term is weighted more heavily than one matching a common term." IDF measures something different: not how often the term appears in ONE document, but how common or rare that term is ACROSS THE ENTIRE COLLECTION of documents being searched. WHY TF ALONE IS AN INCOMPLETE SIGNAL ------------------------------ If relevance scoring used TF alone, a document repeating a very common, low-information word many times (even after stop-word removal per search1-4, some genuinely common words still remain, like "system" in a tech catalog) could score highly purely because that word happens to be common across many documents — without that repetition actually telling you anything distinctive about THIS particular document relative to the others. A search term that appears in nearly every document in the collection provides very little power to distinguish between documents at all, no matter how many times any single document repeats it. WHY COMBINING TF AND IDF PRODUCES A BETTER SIGNAL ------------------------------ Per this chapter, "combining the two: a TF-IDF score weights a document's own term frequency down for common terms and up for rare ones — the classic, foundational formula underlying modern information retrieval." By multiplying (or otherwise combining) TF with IDF, a document's own high term frequency for a RARE, distinctive term gets rewarded heavily (since matching a rare term at all is a strong, useful signal), while a document's high term frequency for a COMMON, widely-shared term gets discounted (since nearly every document could achieve a similar frequency for that same common term, making it a weak, low-information signal). This produces a genuinely more useful ranking than TF alone, because it accounts for how MUCH information a given term match actually conveys about a document's own distinctive relevance, not just how many times it was repeated. WHY THIS WORKS AS AN ANSWER ------------------------------ It defines both TF and IDF individually using the chapter's own wording, and explains specifically why TF alone fails to distinguish distinctive matches from generic repetition, and why combining it with IDF fixes that specific weakness.