Search Implementation and Relevance

A search box looks like a single feature, but behind it sit three systems: an indexing pipeline that keeps a search-optimized copy of your data fresh, a query layer that turns what people type into retrieval and ranking, and a feedback loop that measures whether results are actually good. Most search projects get the first two working and skip the third — and then spend months reacting to complaints that "search is broken" without any way to tell whether a change made it better or worse.

This page covers the end-to-end design: choosing an engine, indexing, analysis and ranking, filters and pagination, semantic and hybrid retrieval, and the evaluation practices that make relevance improve over time.

TL;DR

Quick Example

For modest catalogs, PostgreSQL's built-in full-text search with weighted fields goes a long way:

And the evaluation that tells you whether a ranking change helped — NDCG@10 against human relevance judgments:

Average NDCG over a few hundred representative queries before and after every relevance change.

Core Concepts

Choosing an Engine

See PostgreSQL Full-Text Search, Elasticsearch, and OpenSearch.

Indexing Pipelines

The search index is a derived view. Changes flow from the database through an outbox, CDC stream, or queue into indexing workers that build denormalized documents — product plus category names, prices, inventory, permissions. Plan for full reindexing into a new index behind an alias, so mapping changes and recovery are routine, and monitor indexing lag. See Change Data Capture.

Analysis and Scoring

Text is broken into tokens by analyzers: lowercasing, removing accents, stemming ("running" → "run"), synonyms ("tee" ↔ "t-shirt"), and language-specific rules. Lexical engines score matches with BM25, which rewards rare terms and matches in short fields. Field boosts (title over description), exact-match bonuses, and business signals (popularity, availability, recency) shape ranking. See Elasticsearch Analyzers.

Filters, Facets, and Pagination

Filters (category, price range, in stock, permissions) restrict results without changing scores and are cached by engines. Facets show counts per value so users can refine. Paginate deep results with search-after cursors or point-in-time searches; large offsets are slow and inconsistent as data changes. See Pagination, Filtering, and Sorting.

Query Assistance

Semantic and Hybrid Retrieval

Embeddings capture meaning beyond shared words — "laptop for video editing" can match a product described as "creator workstation". Hybrid search runs lexical and vector retrieval and merges results (for example with reciprocal rank fusion), keeping exact-match precision while adding semantic recall. A reranker — a cross-encoder or learned ranking model — then reorders the top candidates. See Hybrid Search and Reranking.

Access Control

Search must never reveal what a user cannot open. Index permission attributes (tenant, groups, visibility) and apply them as filters on every query, and test this with multiple users. Post-filtering results after retrieval leaks counts and facets.

The Relevance Feedback Loop

A practical loop: sample real queries from logs (head, torso, and tail), grade results with domain experts, evaluate every ranking change offline, A/B test promising changes, and add newly discovered failures to the judgment set.

Best Practices

Treat the Index as Rebuildable

Version index mappings, rebuild into new indices behind aliases, and keep reindexing fast enough to run whenever analysis changes.

Analyze Real Queries First

Look at the top queries, zero-result queries, and long-tail samples before tuning. Real queries rarely look like the ones developers test with.

Change One Thing at a Time

Evaluate each analyzer, boost, or model change against the same judgment set, so you know what helped.

Make Business Rules Explicit

Merchandising boosts and pinned results should be configured and documented, not hidden in ad hoc query code.

Common Mistakes

Tuning Without Measurement

Deep Offset Pagination

Filtering Permissions After Retrieval

FAQ

Should I use PostgreSQL or Elasticsearch for search?

PostgreSQL full-text search is often enough for modest data volumes and simple relevance, with the advantage of no extra system. Elasticsearch or OpenSearch suits large datasets, advanced analysis, facets at scale, and hybrid search.

How do I measure search relevance?

Build judgment lists of queries with graded relevant results, compute offline metrics such as NDCG@10 for ranking changes, and track online signals such as zero-result rate, click-through, and query reformulations.

What is hybrid search?

Combining lexical retrieval (keyword matching with BM25) and semantic retrieval (vector similarity over embeddings), then merging their results, so search handles both exact terms and meaning.

How do I keep the search index in sync with the database?

Publish changes through an outbox or change data capture into indexing workers, monitor indexing lag, and support full reindexing into a new index behind an alias for recovery and mapping changes.

Why do some searches return no results?

Common causes are missing synonyms, overly strict matching, misspellings, analysis that does not match how users write, or genuinely missing content. Track zero-result queries and review them regularly.

Related Topics

References