Relevance Scoring & Full-Text Search Done Right

This is the chapter that formally closes the loop postgres1-6 deliberately left open — a real, in-depth comparison against a purpose-built relevance-ranking algorithm, not just a named category.

The Question Every Search Engine Has to Answer

Given multiple documents that all match a query, how does the engine decide which are the best matches, and in what order to return them? This is exactly the question search1-4's own postings-list frequency and position data was collected for, and exactly what search1-5's own must-vs-filter scoring distinction was building toward.

TF-IDF — The Classic Foundation

Term Frequency (TF): how many times does the search term appear in this document — a document mentioning "wireless" five times is, all else equal, probably more relevant to a "wireless" search than one mentioning it once.

Inverse Document Frequency (IDF): how rare is this term across the whole index — a term appearing in nearly every document tells you less about relevance than a term appearing in only a few. A document matching a rare term is weighted more heavily than one matching a common term.

Combining the two: a TF-IDF score weights a document's own term frequency down for common terms and up for rare ones — the classic, foundational formula underlying modern information retrieval.

BM25 — The Modern Default

BM25 (Best Match 25) has been Elasticsearch/OpenSearch's own actual default scoring algorithm since version 5 — built on the same underlying TF/IDF ideas, with two real, meaningful refinements:

• Term frequency saturation — repeating a term more and more within a document gives diminishing returns to the score, rather than scaling linearly forever. A document that says "wireless" 50 times isn't 50 times as relevant as one that says it once — BM25 accounts for this.
• Field length normalization — a term match in a genuinely short document or field is weighted more heavily than the identical match in a very long one, since a short document mentioning a term is proportionally more "about" that term.

Formally Delivering on postgres1-6's Own Deferred Comparison

postgres1-6 covered ts_rank() as Postgres's own relevance-ranking function, and explicitly deferred a full comparison to this exact course. Here it is, honestly: ts_rank() does incorporate real signals — term-frequency-adjusted weighting via setweight(), some proximity awareness — but it's a genuinely simpler, less sophisticated ranking function than BM25. It doesn't implement the same level of length normalization or frequency saturation refinement, and Postgres's own full-text search generally wasn't designed to be tuned to the same degree. This isn't "Postgres's own version is broken" — it's a genuine, honest "smaller-scale approximation vs. purpose-built, heavily-refined algorithm" difference, matching this course's own consistent pattern of fair, not dismissive, comparisons.

Explaining a Score — The explain API

A genuinely useful, real practical tool: an _explain endpoint shows exactly how a document's own relevance score was computed, term by term — useful for debugging "why did this result rank above that one" in a real, concrete, inspectable way, rather than treating scoring as an opaque black box.

Boosting — Deliberately Tuning Relevance

Fields or query clauses can be given an explicit "boost" multiplier, deliberately increasing their own contribution to the final score — weighting a product's name field higher than its description field, for instance. This is directly comparable in spirit to postgres1-6's own setweight() material (title vs. body weighting).

Hands-On Exercises