Search, Sitemaps and Metadata

A static site has no server to ask, so search has to be a file, and a search engine has to be told, by files and tags, what the site contains. This chapter builds a search that runs in the visitor's browser, a sitemap, one canonical address for every page, and the Open Graph tags that describe a page when it is shared. As in earlier chapters, the results are compared with the Django project on the real content, and two of the differences turned out to be mistakes in the content.

Search That Is a File

There is no search server and no database. When the site is built, an index of every page is written out as one JSON file, /search-index.json. The browser fetches it the first time somebody searches, then looks words up in it. Three small, tested jobs make it work:

1. htmlToText: the words a person would read on a page (tags, scripts and styles removed, character references decoded).
2. buildSearchIndex: for every word, which pages it is on and how often.
3. search: a query against the index.

• Words are lower-cased with their accents taken off, so “Szeretnék” and “szeretnek” are one word.
• Japanese has no spaces, so it is stored as single characters, pairs and groups of three, and a query uses the biggest group that fits. (Taking accents off が would turn it into か, so Japanese is cut out first and never folded.)
• A search needs every word; the last may be only the beginning of a word, so “polite cond” finds “conditional”.
• Typed text is never a query language. It is split into units and looked up in tables, so no input can be an error or an injection.
• Ranking is BM25 (a word counts more when it is rare and frequent, less on a long page) with the title weighted ten to one.
• The index is compact: each page number is written as the difference from the one before, in base 36 (3,7,1:2 is pages 3, 10 and 11, the last with the word twice).

The package has two doors, because a browser must never load code that reads files: the main one (used at build time) and @lw/search/client (prepare and search only), which the page imports.

Is It the Same as Django's?

Step 1: the text

The Django index stores each page's plain text. A fingerprint of each was dumped and compared with the text made here.

The seven pages came from three causes:

• </> (a React fragment written in a code sample) stayed as text here; a parser ignores it.
• ⅖ (two fifths) was a real HTML name my decoder did not know. (Earlier a survey of every page found 57 names that pages use and the shared decoder did not know; it was filled in, and a lower-case fallback was removed because Δ and δ are different letters.)
• Two pages have a mistake in their content: a raw <style> block and a raw <script src> written in a sentence, where <style> was meant. An unclosed <style> or <script> swallows the rest of the page, in a browser and in Python's parser. So a browser hides the rest of both pages, and the Django index holds 524 of 54,617 characters of the Build Tooling page. Here an unclosed one swallows the rest too, so the two agree; the real fix is in the two content files (it is in the review notes).

Step 2: the searches

35 queries were put to the Django search and to this one: words, accents, beginnings, Japanese of 1, 2, 3 and 5 characters, all eight sites, and injection-like text.

The four differences in the sets were: “ありが” found one page too many (with pairs of characters a page that has the first pair in one place and the second in another counts, so groups of three were added to the index), and three queries that found the Build Tooling page, which Django could not (the swallowed text above). The order differs because SQLite and this code compute BM25 a little differently. I stopped there on purpose: matching another engine's order to the last result is not the goal; the same pages, in a sensible order, is.

A search takes a median of 0.04 ms (at worst 1.7 ms) once the index is loaded. It is fetched once per visit, and only when somebody searches.

The Search Page

A client component reads ?q=, fetches the index once, searches it, and shows 20 results at a time. A plain form in the header goes to it, so it works without any script. The page is marked noindex (a list of results is not something to find in a search engine). In a headless Chrome, 18 checks passed:

• The same counts as the Django search (18 for “szeretnek” and for “Szeretnék”, 1 for “brotchen”, 10 for “polite cond”, 25 for “ありがとう” and “ありが”, 5 for “水”, 0 for nonsense).
• An injection-like query is a search with no results. HTML typed into the box (an <img onerror>) is shown as text and nothing runs.
• “the” gives 348 results in pages of 20 (the last page shows 8; a page number past the end shows none, without a crash).
• The header box with accented text finds pages; clicking a result moves to it without a reload; the index is fetched once.
• From opening the page to seeing results: 64 to 136 ms after the first search, which took 439 ms because it loads the index.

One Address for Every Page

The sitemap, the canonical link and the Open Graph address all come from one function, so they cannot disagree. Decision: the canonical form has no trailing slash (Next's way), where the Django project used one. Old addresses with a slash are redirected (308), so no link anyone made is lost, and each page has exactly one address:

The built site was checked page by page (check-seo.mjs reads every built page):

Can the check fail? Because the sitemap and the canonical link normally use the same function, it takes a change to one of them to see it work: giving only the canonical link a trailing slash made all 561 pages fail. Note that robots.txt is decided when the site is built here (set LW_ALLOW_CRAWLING=1 for the real build); the Django project decides when it is asked.

Hands-On Exercises