Agentic RAG & Advanced Retrieval: Hybrid Search, Reranking, GraphRAG

Agentic RAG & Advanced Retrieval

Basic retrieval-augmented generation (RAG) — embed the question, fetch the top-k similar chunks, stuff them in the prompt — gets you surprisingly far, and then hits a wall. Real questions are ambiguous, span multiple documents, need fresh data, or require reasoning across facts that no single chunk contains. Advanced retrieval and agentic RAG are the techniques that push past that wall.

The shift is from RAG as a fixed pipeline (retrieve once, then generate) to RAG as a dynamic process the model participates in — deciding what to search for, searching multiple times, reranking what comes back, and iterating until it has what it needs. This page covers the toolkit: better retrieval mechanics (hybrid search, reranking, query rewriting), structural approaches (GraphRAG), and the agentic pattern that ties them together.

TL;DR

Basic RAG (embed → top-k → generate) fails on ambiguous, multi-hop, or freshness-sensitive questions. Advanced retrieval fixes the retrieval quality; agentic RAG fixes the process.
Hybrid search combines semantic (vector) and keyword (lexical/BM25) search — catching both meaning and exact terms (names, codes, jargon) that pure vectors miss.
Reranking re-scores retrieved candidates with a more precise (cross-encoder) model, so the best chunks land at the top of a small context budget.
Query rewriting transforms the raw question (expand, decompose, add context) before searching — the retriever's input matters as much as the index.
GraphRAG builds a knowledge graph over your corpus so the system can answer questions that require connecting facts across documents.
Agentic RAG lets the model plan and iterate: choose tools, search multiple times, judge sufficiency, and refine — retrieval becomes a loop, not a single step.

Quick Example

Agentic retrieval in its simplest form: give the model a search tool and let it decide when and what to search, iterating until it can answer. This is RAG as a loop, not a fixed pipeline:

Faced with a comparison question, the model can search 2024 policy, then 2026 policy, then synthesize — something a single top-k fetch can't do.

Core Concepts

Why Basic RAG Hits a Wall

A single embed-and-fetch step assumes the answer lives in a few chunks that are semantically similar to the question as asked. That breaks down when:

The question is ambiguous or underspecified — the raw query retrieves the wrong neighborhood.
The answer needs multiple hops — "compare X and Y," "what changed between A and B" requires assembling facts from different places.
Exact terms matter — product codes, error IDs, names, and rare jargon are where semantic search is weakest and keyword search shines.
One retrieval isn't enough — the model needs to see initial results before it knows what to search for next.

Each advanced technique targets one of these failure modes.

The Retrieval Quality Stack

Before adding agentic loops, fix the retrieval itself:

Advanced Retrieval Techniques

Hybrid Search

Semantic (vector) search captures meaning — "car" matches "automobile." Keyword search (BM25/lexical) captures exact tokens — the error code E-4021, a person's name, a specific SKU. Each fails where the other succeeds. Hybrid search runs both and fuses the scores (commonly with reciprocal rank fusion), so you retrieve on meaning and exact terms. For most real corpora — full of names, IDs, and jargon — hybrid beats pure vector search noticeably.

Reranking

Vector top-k is a coarse filter: fast, but it puts merely-related chunks near the top. A reranker is a more expensive, more accurate model (typically a cross-encoder that reads the query and each candidate together) that re-scores a retrieved shortlist. The pattern: retrieve 50 candidates cheaply, rerank to the best 5, send those. This gets the most relevant context into a small token budget — a big quality lever for modest cost.

Query Rewriting and Decomposition

The retriever's input is the query, and the raw user question is often a poor query. Techniques:

Rewriting/expansion — clean up and enrich the query with synonyms or context ("it" → the actual subject from the conversation).
Decomposition — split a complex question into sub-questions, retrieve for each, and combine ("compare A and B" → retrieve A, retrieve B).
Hypothetical answers (HyDE) — generate a hypothetical answer and search with that, since an answer often embeds closer to the source than the question does.

GraphRAG

Vector RAG treats documents as a bag of independent chunks; it can't easily answer "what connects these entities?" GraphRAG builds a knowledge graph — entities and their relationships extracted from the corpus — and retrieves over that structure. This excels at multi-hop and global questions ("summarize the themes across all incident reports," "how are these three teams related?") that require connecting information no single chunk holds. The cost is building and maintaining the graph, so use it when relational, cross-document reasoning is the actual need.

Agentic RAG

The unifying idea: stop treating retrieval as a fixed step and let the model drive it. In agentic RAG, retrieval is a tool (or several) the model calls as part of a reasoning loop:

This gives the model the ability to search multiple times, choose which source to query, decompose hard questions itself, judge whether it has enough, and recover from a bad first retrieval. It naturally extends to multiple tools — a vector store, a keyword index, a SQL database, a web search — with the model routing between them. Agentic RAG is where RAG meets agents: the retrieval strategy is emergent, not hardcoded.

💡 Tip: Agentic RAG is more capable and more expensive and slower — each iteration is another model call and more context. Start with strong single-pass retrieval (hybrid + rerank), and add agentic loops only for questions that genuinely need multi-step retrieval.

Best Practices

Fix Retrieval Before Adding Agents

Most "RAG is bad" problems are retrieval-quality problems, not reasoning problems. Add hybrid search and a reranker first — they're cheap, single-pass, and often the biggest quality jump. Reach for agentic loops only when single-pass retrieval genuinely can't gather what a question needs.

Measure Retrieval and Generation Separately

A RAG system fails at retrieval (wrong chunks came back) or generation (right chunks, bad answer). Measure each: retrieval metrics (recall, precision of the fetched chunks) apart from answer metrics (faithfulness, correctness). See LLM Evaluation. A single end-to-end score hides where it's broken.

Match the Technique to the Failure Mode

Don't adopt GraphRAG because it's fashionable — adopt it if your questions are multi-hop and relational. Don't add agentic loops if single-pass works. Each technique targets a specific failure; diagnose the failure first, then apply the matching fix.

Cap the Agentic Loop

An agentic retriever can search indefinitely. Set an iteration limit, budget the context (each pass adds tokens), and monitor cost per query. See Context Engineering.

Keep Grounding and Guardrails

Advanced retrieval still feeds untrusted content into the model. Retrieved chunks can carry prompt injection; frame them as data, and validate the final answer against the sources for faithfulness.

Common Mistakes

Reaching for Agentic RAG to Fix Bad Chunks

Pure Vector Search for Everything

Never Reranking

Sending the raw vector top-k straight to the model wastes context on mediocre chunks. A reranker on a 50→5 shortlist is one of the cheapest, highest-impact upgrades available.

FAQ

What's the difference between RAG and agentic RAG?

Basic RAG is a fixed pipeline: retrieve once, then generate. Agentic RAG makes retrieval a tool the model controls inside a reasoning loop — it can search multiple times, refine its queries, choose among sources, judge whether it has enough, and recover from a bad first search. It's more capable for complex, multi-hop questions but slower and more expensive per query.

Do I need hybrid search or is vector search enough?

Pure vector search captures meaning but is weak on exact terms — names, product codes, error IDs, rare jargon — which real corpora are full of. Hybrid search blends semantic and keyword (BM25) scoring so you get both, and it usually beats vector-only noticeably. It's a cheap, high-value default; add it before more exotic techniques.

What does a reranker actually do?

A reranker re-scores a shortlist of retrieved candidates using a more accurate (and more expensive) model — typically a cross-encoder that reads the query and each candidate together, rather than comparing precomputed vectors. You retrieve many cheaply, then rerank to the best few, getting the most relevant chunks into a small token budget. It's one of the highest-impact, lowest-effort RAG upgrades.

When should I use GraphRAG?

When your questions require connecting facts across many documents — multi-hop reasoning ("how are these entities related?") or global synthesis ("what are the themes across all these reports?") — which chunk-based vector RAG struggles with. GraphRAG builds a knowledge graph of entities and relationships to answer those. It costs more to build and maintain, so use it when relational, cross-document reasoning is the real need, not by default.

How do I know if my RAG problem is retrieval or generation?

Evaluate the two stages separately. Check retrieval quality — did the right chunks come back? — apart from generation quality — given good chunks, was the answer faithful and correct? If the right context isn't being retrieved, no amount of prompt tuning helps; fix retrieval (hybrid, rerank, query rewriting). If retrieval is good but answers are wrong, focus on the prompt and grounding.