Recommendation Systems & Ranking Evaluation

Recommendation Systems and Ranking Evaluation

Recommendations decide much of what people see online — products, videos, articles, songs, jobs, people to follow. A good recommender helps users find things they would not have searched for; a bad one shows the same popular items to everyone, reinforces its own past mistakes, or looks impressive in offline metrics and does nothing in production. The difference lies less in clever models than in solid baselines, honest evaluation, and awareness of the feedback loops recommendations create.

Most production systems share one architecture: retrieve a few hundred candidates from millions, rank them with a richer model, then re-rank for diversity, freshness, and business rules. This page covers each stage and, above all, how to evaluate them.

TL;DR

Most signals are implicit (clicks, purchases, watch time), not explicit ratings — and absence of interaction is not dislike.
Start with baselines: popularity, recently popular, and item-to-item co-occurrence. They are hard to beat and essential for comparison.
Scale with collaborative filtering (matrix factorization) or two-tower embedding retrieval plus approximate nearest-neighbor search.
Rank candidates with feature-rich models, then re-rank for diversity, freshness, and rules.
Handle cold start with popularity, content features, and onboarding signals.
Evaluate offline with time-based splits and recall@k, NDCG@k, coverage, and diversity; confirm with online A/B tests, mindful of position and exposure bias.

Quick Example

An item-to-item baseline with a popularity fallback, evaluated with a time-based split:

Every more sophisticated model should be compared against numbers like these on the same split.

Core Concepts

Feedback Data

Define the objective carefully: clicks are easy to optimize and easy to game; purchases, long-term retention, or satisfaction are harder but more meaningful.

Baselines

Popularity (overall and recent) — surprisingly strong and the default for new users.
Item-to-item similarity from co-occurrence ("people who bought this also bought") — simple, explainable, and robust.
Content similarity from item embeddings or attributes — works without interaction data.

Collaborative Filtering and Embeddings

Matrix factorization (for example alternating least squares for implicit feedback) learns user and item vectors whose dot products predict interactions. Two-tower models learn user and item embeddings with neural networks from features and histories, so item vectors can be indexed for approximate nearest-neighbor search and retrieved in milliseconds. See Embeddings and Vector Databases.

Retrieval, Ranking, and Re-Ranking

Retrieval optimizes recall (do good items reach the ranker?); ranking optimizes the order of the top positions. See Personalization.

Cold Start

New users: popular and trending items, onboarding questions, context such as location or referral source, and fast adaptation from first clicks.
New items: content-based similarity, exploration slots that give items exposure, and metadata-driven embeddings.

Evaluating Recommendations

Offline Evaluation

Split data by time — train on the past, evaluate on the future — rather than randomly, which leaks future behavior into training.

Report metrics by segment — new versus returning users, popular versus long-tail items — not just overall.

Bias and Feedback Loops

Users can only interact with what they are shown, and they click top positions more regardless of quality (position bias). Logged data therefore reflects the old system's choices (exposure bias), and models trained on it tend to reinforce popular items. Mitigations include logging impressions and positions, randomized exploration, inverse propensity weighting, and evaluating with counterfactual or interleaving methods.

Online Evaluation

Offline improvements often fail to translate online. A/B tests measure what matters: engagement, conversion, retention, and revenue, with guardrail metrics such as complaints, diversity, and latency. Interleaving compares rankers with far fewer users for ranking changes. See A/B Testing.

Serving

Precompute candidates for heavy users and popular items; compute personalized results at request time where freshness matters.
Feature stores keep training and serving features consistent. See Feature Engineering.
Latency budgets of tens of milliseconds constrain model size; use ANN indexes and caching.
Monitoring: watch distribution shifts, coverage, empty results, and model staleness.

Best Practices

Beat the Baseline Before Shipping

Any new model must outperform popularity and item-to-item baselines on time-based splits, by segment, before an A/B test.

Log Impressions, Not Just Clicks

Record what was shown, where, and why, so training data and evaluation can account for exposure and position.

Optimize for Long-Term Value

Prefer objectives tied to satisfaction and retention over raw clicks, and include guardrails against clickbait and filter bubbles.

Keep Humans in the Loop

Review recommendations for sensitive categories, fairness, and quality regularly; metrics miss embarrassing or harmful results.

Common Mistakes

Random Train-Test Splits

Optimizing Clicks Alone

Ignoring Cold Start

FAQ

What is a recommendation system?

A system that predicts which items — products, content, people — a user is likely to find valuable and presents a ranked selection, using interaction data, item and user features, and context.

What is collaborative filtering?

A family of methods that recommend items based on patterns of interactions across many users — for example, items liked by similar users or items frequently consumed together — without needing item content.

How do I evaluate a recommender offline?

Split interaction data by time, generate recommendations from training data, and measure recall@k, NDCG@k, coverage, and diversity on later interactions, by user segment, compared with baselines.

What is the cold start problem?

The difficulty of recommending for new users or new items with little or no interaction history. It is addressed with popularity and content-based methods, onboarding signals, context, and exploration.

Why do offline and online results disagree?

Logged data reflects what the previous system showed (exposure and position bias), offline metrics do not capture user experience fully, and real behavior changes when recommendations change. Online A/B tests remain the final judge.