Search and Sitemaps

With 4,400 pages, a visitor who knows what they want should not have to click through menus to find it, and a search engine should be told what each site contains. This chapter adds a search box that only searches the current site, a sitemap for each site, one canonical address for every page, and a robots.txt that keeps crawlers out until launch.

Choosing a Search Engine

The Index

The page body is stored as HTML, so the importer first turns it into plain text (tags removed, <script> and <style> ignored, a space at every block tag so table cells do not join into one word). The text goes into two FTS5 tables, each using the page's own id as its row id:

The site column is stored but not searched, so one table serves all eight sites and each search filters on it. The importer keeps the index in step: it re-indexes new and changed pages after saving them and removes pages from the index before pruning them. The importer's version number went up, so the first import after this chapter re-imported every page once.

Real import: 4,406 pages updated and 2 created in 38 seconds, no errors, 4,408 rows in each index table (421 in the Japanese one). The database grew from about 99 MB to 162 MB after a VACUUM, so the indexes cost roughly 63 MB.

Turning Typed Text into a Query

FTS5 has its own query language: quotes, AND, OR, NEAR, brackets, a leading minus, column filters like title:x. Text typed by a visitor that is not valid in it is an error, and text that is valid can change the meaning. So the visitor's text is never put in as it is. Only word characters are kept, each word is quoted, at most eight are used, and the last one may be the start of a word:

Results are ranked with bm25, with a title match worth ten times a body match. Each result carries a snippet. The match is wrapped in control characters (which cannot appear in page text), the snippet is HTML-escaped, and only then are the markers turned into <mark> tags, so page text that contains a script is shown as text.

Japanese

The trigram table finds any run of three or more characters. One or two characters (水, 水を) are too short for it, so those are answered by scanning the 421 Japanese pages with LIKE, with % and _ escaped. That scan took 5 ms; it will slow down if the number of Japanese pages grows a lot.

Real Searches

A search on one site never returns another site's pages: the site filter is part of every query. The search box sits in every site's header and goes to that site's own /search/. Results come twenty at a time with Previous and Next links, and the empty search shows a short help line. The page was checked in headless Chrome: the matched word is highlighted inside each snippet.

The PostgreSQL Version (not run)

If the database ever becomes PostgreSQL, the usual route is a generated SearchVectorField with a GIN index, queried with SearchQuery and ranked with SearchRank. The FTS5 migration and functions here do nothing on other databases (they check the database vendor), so the switch would replace this app's two modules, not the rest of the project. This was not run here.

Sitemaps

A sitemap is a list of a site's pages for search engines. Each site has its own at /sitemap.xml, built with Django's django.contrib.sitemaps from that site's pages only, with each page's own address:

• The limit is 50,000 URLs per sitemap; the largest site has 1,376, so one file is enough.
• A lastmod date appears only where the page's banner has a date. No date is invented, because a wrong date teaches a crawler to ignore the field.
• Addresses with non-ASCII characters are percent-encoded (hiragana_%E3%81%82/).
• Behind the proxy the addresses are https, because the proxy settings from Chapter 2 tell Django the original request was secure.

One Address per Page

Every page now has a <link rel="canonical"> and a <meta name="description"> (from the banner's Topic line, or the first 160 characters of text). The important rule is that the sitemap and the canonical link get their address from the same function, so they cannot disagree. This was checked on real pages: 75 sampled pages on the languages site (including ones with Japanese in the address) and 60 each on programming and ai, and every page's canonical link was in its site's sitemap. Listing pages and the front page do not have a canonical link yet.

robots.txt

Each host answers its own /robots.txt. Until launch it says Disallow: /; setting LW_ALLOW_CRAWLING=1 switches it to Allow: / plus a Sitemap: line naming that site's own sitemap. Putting it behind a setting makes opening the sites to search engines one deliberate line on the launch checklist.

Hands-On Exercises