Exercise 1: Why Crawling Doesn't Guarantee Indexing — Possible Solution ==================================================================== WHAT CRAWLING ACTUALLY ACCOMPLISHES ------------------------------ Per this chapter, crawling is the discovery step — a bot follows links or reads a sitemap and fetches a page's content. That's the entire job of this stage: find the page, retrieve what's on it. Nothing about crawling itself involves a judgment call about whether the page is worth storing or serving in results. WHAT INDEXING ADDS ON TOP ------------------------------ Per this chapter's own table, indexing is a separate step where a crawled page's content "is processed and stored in the search engine's own massive index." This is where an actual evaluation happens — the search engine decides whether this specific page is worth adding to the searchable index at all, not merely whether it could be reached. A PLAUSIBLE REAL REASON A CRAWLED PAGE MIGHT NOT BE INDEXED ------------------------------ Thin or duplicate content, per this chapter's own example, is a realistic case: a page that's been crawled successfully but contains very little unique substance — say, a near-identical copy of content that already exists elsewhere on the same site, or a page with almost no real text at all. A search engine can determine this only after actually reading the page's content, which requires having crawled it first. The crawl succeeds (the bot reaches and reads the page); the indexing decision then separately concludes there's nothing distinct enough here to justify adding it to the index. WHY THIS PROVES CRAWLING AND INDEXING ARE GENUINELY SEPARATE STAGES ------------------------------ If crawling automatically implied indexing, there would be no meaningful distinction between the two stages at all — every fetched page would simply be in the index by definition. The fact that a page can be crawled and then deliberately excluded from the index (thin content, being one real example) is direct evidence that these are two independent decisions, made at two different points, using different criteria: crawling asks "can this be reached," indexing asks "is this worth keeping." WHY THIS WORKS AS AN ANSWER ------------------------------ It explains what each stage actually evaluates using this chapter's own definitions, and gives a concrete, plausible real-world scenario (thin or duplicate content) where a page clears the crawling bar but fails the separate, later indexing judgment.