Technical SEO: Crawlability, Indexability & Sitemaps

SEO Fundamentals

Chapter 5 · Technical SEO: Crawlability, Indexability & Sitemaps

This is the course's own central technical chapter — where seo1-1's own crawling and indexing stages get their real, concrete tools.

robots.txt — The Other Side of dsproj1-1's Own Warn-Box

# /robots.txt
User-agent: *
Disallow: /admin/
Disallow: /tmp/
Allow: /

Sitemap: https://example.com/sitemap.xml

The mirror image of a chapter this site already has

dsproj1-1 taught what a scraper must respect when it encounters a site's own robots.txt — checking it before crawling, honoring its disallow rules. This chapter teaches the other half: what a site owner actually writes there. Same file, same protocol, opposite side of the exact same interaction.

robots.txt is a request, not a lock — well-behaved crawlers (including Googlebot) honor it, but nothing technically prevents a page from being fetched by something that ignores it. It manages crawl behavior for cooperative bots; it is not a security or access-control mechanism.

Sitemaps — Making Pages Findable Without Relying on Links Alone

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/japanese/hiragana</loc>
    <lastmod>2026-07-17</lastmod>
  </url>
</urlset>

Why a sitemap matters even on a well-linked site

seo1-1's own crawling stage relies primarily on following links. A sitemap is the explicit backup — a direct, machine-readable inventory of every URL that exists, submitted straight to a search engine, regardless of how well internal linking happens to be structured. Pages that are genuinely orphaned from a site's own navigation are exactly the case a sitemap is built to rescue.

Canonical Tags — Telling a Search Engine Which Version Is Real

<link rel="canonical" href="https://example.com/japanese/hiragana">

The same content is often reachable at more than one URL — with and without a trailing slash, with a tracking parameter, over HTTP versus HTTPS. Without a canonical tag, a search engine has to guess which version is the "real" one to index and rank, and duplicate versions can dilute ranking signals that should have been concentrated on a single URL. The canonical tag states the answer directly.

noindex — Removing a Page From the Index on Purpose

<meta name="robots" content="noindex">

Tool	What it actually does
robots.txt Disallow	Asks crawlers not to fetch the page at all
noindex meta tag	Allows crawling, but asks the page not to be added to the index

A genuinely common mistake: blocking a page in robots.txt and also adding a noindex tag. Since robots.txt already prevents the page from being crawled, the crawler never sees the noindex instruction at all — the two tools solve different problems and using them together on the same page is usually redundant at best, contradictory at worst.

Mobile-First Indexing

Search engines now primarily use a site's mobile version to crawl and index content, not the desktop version — meaning a page whose mobile layout hides or omits content the desktop version shows is effectively invisible for that missing content, indexing-wise. This is fundamentally a crawling/indexing concern, not a new design topic — the actual responsive-design techniques that solve it are css_beginner_11's own material in full; nothing here needs re-teaching, only applying with this specific consequence in mind.

Hands-On Exercises

Exercise 1

Explain why robots.txt is described as "a request, not a lock," and what this implies about using it as a security measure.

📄 View solution

Exercise 2

Explain the specific difference between robots.txt Disallow and a noindex meta tag, and why combining both on the same page is usually a mistake rather than "extra safety."

📄 View solution

Exercise 3

Explain why a sitemap still matters on a site with good internal linking, and describe the specific kind of page a sitemap is built to rescue.

📄 View solution

Chapter 5 Quick Reference

robots.txt — the site-owner side of dsproj1-1's own scraper-side warn-box; a request cooperative crawlers honor, not an access-control mechanism
Sitemaps — an explicit URL inventory, the backup for pages links alone might not surface
Canonical tags — resolve duplicate-URL ambiguity so ranking signals concentrate on one real version
noindex — allows crawling, blocks indexing; combining with robots.txt Disallow is usually redundant or contradictory
Mobile-first indexing — a crawling/indexing consequence, solved with css_beginner_11's own already-complete responsive-design techniques
Next chapter: Site Performance as a Ranking Factor