Text Classification & Retrieval Baselines (TF-IDF, BM25)

Text Classification and Retrieval Baselines

It is tempting to start every text problem with an embedding model or a large language model. But simple, transparent baselines — TF-IDF features with a linear classifier, or BM25 for search — often reach most of the achievable accuracy, train in seconds, cost almost nothing to run, and are easy to explain. More importantly, without a baseline you cannot tell whether a sophisticated model is actually better, or whether its gains justify its cost, latency, and opacity.

A good baseline is also a diagnostic tool. Its errors reveal label noise, ambiguous categories, and leakage; its strongest features show what the model is really keying on; and comparing it across languages, customer segments, and time periods shows how robust the task is. This page covers building classification and retrieval baselines and evaluating them honestly.

TL;DR

Classification baseline: TF-IDF (word and character n-grams) + logistic regression or a linear SVM.
Retrieval baseline: BM25, which remains competitive across many domains and complements dense retrieval in hybrid search.
Split data stratified by label, and by time or source when the model will face new periods or customers; avoid leakage from duplicates.
Report per-class precision, recall, and F1, macro averages for imbalanced labels, and recall@k / MRR / nDCG for retrieval.
Do error analysis and inspect top features to find label noise and spurious cues.
Check distribution shift: language, label mix, domain, and time — and compare advanced models against the baseline on the same splits.

Quick Example

A support-ticket classifier baseline with scikit-learn:

A BM25 retrieval baseline evaluated with recall@10:

Core Concepts

Classification Baselines

Character n-grams (char_wb) handle misspellings, morphology, and languages without whitespace tokenization better than words alone; combining both is a cheap win. Class weights help with imbalanced labels.

Retrieval Baselines

BM25 scores documents by term overlap with the query, weighted by inverse document frequency and normalized by document length. It needs no training data, explains its matches, and on benchmarks such as BEIR remains a strong baseline across domains where dense retrievers trained elsewhere can underperform. Use the same tokenization and analyzers you would use in production (stemming, stop words, language analyzers), and combine BM25 with dense retrieval in hybrid search when vocabulary mismatch matters. See Hybrid Search and Embeddings.

Honest Evaluation Splits

Random splits over templated or duplicated text inflate scores dramatically. See Data Leakage and Validation Splits.

Metrics

Choose metrics that match costs: missing a fraud ticket may matter more than mislabeling a billing question. See Model Evaluation.

Robustness and Distribution Differences

Slicing results this way often reveals that a model with a good overall score fails badly for one language or segment.

Error Analysis

Read misclassified examples grouped by true and predicted label. Common findings: inconsistent labels between annotators, overlapping categories that should be merged, spurious cues (a signature or template phrase that correlates with a label), and genuinely ambiguous texts. Fixing data and label definitions usually helps more than changing models.

Best Practices

Always Build the Baseline First

A one-hour TF-IDF or BM25 baseline sets the bar that every complex model must clearly beat.

Evaluate on the Data the Model Will Face

Use time, group, and deduplicated splits that mirror production, and report per-slice metrics.

Inspect Features and Errors

Top features and error samples reveal leakage, spurious correlations, and labeling problems early.

Compare Cost, Not Only Accuracy

Weigh accuracy gains of advanced models against latency, cost, explainability, and maintenance.

Common Mistakes

Leakage Through Duplicates

Reporting Only Accuracy

Skipping the Baseline

FAQ

Why use TF-IDF baselines when embeddings and LLMs exist?

They are fast, cheap, interpretable, and often surprisingly accurate, and they set the bar that more complex models must beat by enough to justify their cost.

Is BM25 still useful with vector search?

Yes. BM25 handles exact terms, codes, and names well and is robust across domains; combining it with dense retrieval in hybrid search usually beats either alone.

What is the difference between macro and micro F1?

Macro-F1 averages F1 across classes equally, highlighting performance on rare classes. Micro-F1 aggregates all predictions, so frequent classes dominate.

How should I split text data for evaluation?

Stratify by label, deduplicate near-identical texts, and split by time or source when the model will face new periods or customers.

What metrics should I use for retrieval?

Recall@k for whether relevant documents are found, MRR for how high the first relevant result ranks, and nDCG@k when relevance is graded.