Exercise 1: Why robots.txt Is "A Request, Not a Lock" — Possible Solution ==================================================================== WHAT ROBOTS.TXT ACTUALLY DOES, MECHANICALLY ------------------------------ Per this chapter, robots.txt is a plain text file a site publishes at a known location, listing rules about which paths crawlers should or shouldn't fetch. Per this chapter, "well-behaved crawlers (including Googlebot) honor it" — the key word being "honor." The file itself has no technical enforcement mechanism attached to it; it doesn't block requests, encrypt content, or require authentication. It's purely informational. WHY THIS MAKES IT A REQUEST RATHER THAN AN ENFORCEMENT MECHANISM ------------------------------ Per this chapter, "nothing technically prevents a page from being fetched by something that ignores it." A crawler that simply doesn't bother checking robots.txt, or that deliberately disregards its rules, faces no technical obstacle actually retrieving the disallowed pages — the server will respond to the request exactly as it would to any other, since robots.txt compliance is entirely voluntary on the crawler's own part, not enforced by the web server itself. WHY THIS MATTERS FOR WHAT robots.txt CAN AND CANNOT BE USED FOR ------------------------------ Per this chapter, "it manages crawl behavior for cooperative bots; it is not a security or access-control mechanism." Genuinely sensitive content — anything that actually needs to stay hidden from anyone who might try to access it — requires real access control (authentication, server-side permission checks) rather than a polite request in a text file that only well-behaved actors will respect. WHAT USING IT AS A SECURITY MEASURE WOULD GET WRONG ------------------------------ Treating a robots.txt disallow rule as equivalent to actually protecting a page would be a real, exploitable mistake — anyone curious enough to look could simply read the robots.txt file itself (it's a plain, publicly accessible text file by design) to discover exactly which paths the site owner didn't want indexed, and then visit those paths directly with a tool that ignores the file's own instructions. In this sense, publishing overly specific disallow rules for genuinely sensitive paths can actually make them easier to find, not harder. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains the purely voluntary, non-enforced nature of robots.txt compliance using this chapter's own language, and connects that directly to why treating it as a security boundary rather than a crawl-behavior request is a genuine, practical mistake.