Exercise 3: Why Lemmatization Can Do What Stemming Structurally Cannot — Possible Solution ==================================================================== WHAT STEMMING ACTUALLY DOES, PER THIS CHAPTER ------------------------------ Per this chapter's own compare-table, stemming is "crude, rule-based chopping of common suffixes — fast, no dictionary needed." A stemmer works by applying mechanical rules that strip off recognizable endings (like "-ing," "-ed," "-s") from a word, with no actual knowledge of what the word means or how it relates to other words — it's purely a pattern-matching operation on the word's own letters. WHAT LEMMATIZATION ACTUALLY DOES, PER THIS CHAPTER ------------------------------ Per this chapter's own compare-table, lemmatization is "dictionary/ grammar-aware reduction to a genuine root word — slower, more accurate," with the example "'better' → 'good.'" Lemmatization consults an actual dictionary/grammatical knowledge base that knows, as a specific fact, that "better" is the comparative form of the adjective "good" — a genuine linguistic relationship, not a matter of shared letters or a common suffix. WHY "BETTER" → "GOOD" IS STRUCTURALLY IMPOSSIBLE FOR STEMMING ------------------------------ "Better" and "good" share no common letters, no common suffix, and no recognizable pattern a suffix-stripping rule could ever connect them through — there is no mechanical transformation (removing "-ing," "-ed," or any similar ending) that turns the letters "b-e-t-t-e-r" into "g-o-o-d." A rule-based stemmer, which per this chapter's own description only ever chops off recognizable letter patterns, has no possible mechanism for making this specific connection, no matter how the rules are tuned — the relationship between these two words is a fact about English grammar (comparative-to-base-adjective mapping), not a fact about shared spelling. WHAT STEMMING WOULD MOST PLAUSIBLY PRODUCE FOR "BETTER" INSTEAD ------------------------------ Given that a rule-based stemmer only strips recognizable suffixes and "better" doesn't end in any of the common suffix patterns a stemmer typically targets (no "-ing," "-ed," "-s," etc.), a typical stemmer would most plausibly leave "better" completely unchanged, or at most strip a trailing "-er" (a pattern sometimes associated with comparative forms in stemming rule sets, like "smaller" → "small"), producing something like "bett" — a genuine non-word, exactly the kind of malformed output this chapter's own "running" → "runn" example already demonstrated stemming can produce. WHY THIS CONFIRMS THE STRUCTURAL DIFFERENCE, NOT JUST A QUALITY GAP ------------------------------ This isn't a case of lemmatization simply doing a "better job" at the same underlying task stemming attempts — it's doing a categorically different kind of operation (consulting real grammatical knowledge) that a purely mechanical, letter-pattern-based process has no way to replicate, regardless of how sophisticated its own suffix rules become. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains precisely why "better" and "good" share no letter-level pattern a stemmer could exploit, contrasts this with lemmatization's own genuine grammatical-knowledge basis, and identifies a concrete, plausible stemmer output for "better" that either leaves it unchanged or produces a malformed non-word, consistent with the chapter's own "runn" example.