RAG Ingestion: Index Freshness & Permission-Aware Retrieval

RAG Ingestion Freshness and Access Control

A retrieval-augmented generation system answers from its index, so the index decides what the assistant believes and what it can reveal. If the index still holds last quarter's pricing policy, the assistant cites it confidently. If a document was deleted or its sharing was revoked, the assistant may go on quoting it — possibly to people who were never allowed to read it. Neither failure shows up in a demo; both show up in production.

The index is a derived copy of source systems — wikis, drives, ticketing tools, databases — and carries the same obligations as any copy: stay in sync, honor deletions, and enforce the source's permissions. This guide covers the ingestion side of RAG: keeping the index fresh, keeping it permission-aware, and proving that it does not leak.

TL;DR

Treat the index as a replica of its sources: sync changes incrementally, including updates, deletions, and permission changes.
Detect changes with change feeds or delta APIs, and run periodic full reconciliation to catch anything missed.
Replace a document's chunks when it changes — never append — and re-embed only what actually changed.
Copy permissions into every chunk and filter at query time using identity resolved on the server, never by instructing the model.
Set freshness targets per source, and propagate permission revocations faster than content updates.
Test for leakage with canary documents and cross-user retrieval tests.

Quick Example

One step of an incremental sync: apply changes from a source's change feed, delete removed documents, update permissions without re-embedding when only access changed, and replace chunks when content changed.

At query time, the user's identity and group memberships come from the server's session, and the filter is applied inside the vector search:

Filter syntax varies by vector database; the principle — permissions applied during retrieval, from trusted identity — does not.

Core Concepts

Freshness

Freshness lag is the time between a change in the source and the index reflecting it. Different sources need different targets: a pricing policy or a security advisory may need minutes; an engineering wiki can tolerate hours. Measure lag per source and display "last updated" dates in citations, so users can judge an answer's currency.

Sync Strategies

Most robust pipelines combine fast incremental updates with a periodic full reconciliation: list every document ID and permission set in the source, compare with the index, and fix differences — including orphaned chunks whose documents no longer exist. See Change Data Capture.

Deletions and Superseded Content

Deleted, archived, and unshared documents must disappear from retrieval, not merely stop being updated. Superseded versions — last year's handbook, an old API version's docs — cause subtler errors: the retriever returns both, and the model blends them. Keep only the current version, or tag chunks with effective_from and superseded_by metadata and filter or down-rank accordingly. See RAG Chunking.

Re-Embedding

Hashing document content avoids re-embedding unchanged documents, which keeps sync cheap. Changing the embedding model is different: vectors from different models are not comparable, so build a new index with the new model alongside the old one, validate retrieval quality, and switch over — see Embeddings.

Access Control

Early, Late, and Hybrid Binding

Filtering after retrieval with a small top_k can leave nothing: if eight candidates come back and the user may see none, the answer is empty. Filter inside the search where possible — see Vector Metadata Filtering.

Mapping Source Permissions

Source systems express access differently: drive sharing, wiki space permissions, ticket visibility, nested groups, links shared with "anyone in the organization" or "anyone with the link". Map each to principals your application understands, expand nested groups (or index group IDs and expand them at query time), and decide deliberately how to treat link-based sharing — often by excluding such documents or limiting them to the organization.

What Never Works

Instructing the model not to reveal restricted content. Anything placed in the prompt can appear in the answer; unauthorized text must never reach the prompt.
A service account with access to everything, and no per-user filtering, which makes every user as privileged as the bot.
Client-supplied filters: the browser cannot be trusted to say which groups the user belongs to.

Related Risks

Answer caches must be keyed by permission scope, or one user's cached answer serves another — see Multi-Tenant Application Design.
Retrieval logs contain sensitive passages and need the same protection as the sources.
Ingested content can carry instructions: documents from outside sources may contain text aimed at the model. Treat retrieved text as data, record source trust levels, and see Prompt Injection.

Testing for Leakage

Canary Documents

Place unique marker strings in documents restricted to one group, then query as users outside that group and assert the marker never appears in retrieved chunks or answers:

What to Cover

Users in different groups and tenants retrieving each other's restricted content — the same discipline as Authorization Testing Across Tenant Boundaries.
Permission revocation, document deletion, and moves between restricted locations, each within the freshness target.
Paraphrased and indirect queries, not only exact titles.
The full answer path, including caches and conversation memory, not only the retriever.

Monitor in Production

As a defense in depth, check each retrieved chunk's permissions again when assembling the prompt, and alert if a chunk the user may not see ever reaches that point. It should never happen; if it does, something upstream is broken.

Best Practices

Publish Freshness Like an SLO

Track per-source lag, the time of the last successful sync, and sync error counts on a dashboard, with alerts when targets are missed. Stale knowledge is a reliability problem.

Prioritize Permission Changes

Revocations are more urgent than content edits. Process sharing changes and deletions ahead of routine updates, and give them a tighter target.

Index Less Sensitive Data

Exclude sources or classifications — HR cases, legal matters, credentials — that the assistant has no reason to retrieve. Data that is never indexed cannot leak through retrieval.

Cite Sources With Dates

Show the source and its last-updated date with each answer. Users can then spot stale content, and reported mistakes lead straight to the document that needs fixing.

Common Mistakes

Append-Only Ingestion

Polling That Never Sees Deletions

Access Control in the Prompt

A Bot That Sees Everything

FAQ

How fresh does a RAG index need to be?

As fresh as the cost of a stale answer demands. Policies, prices, and security guidance may need minutes; reference documentation may tolerate hours or a day. Permission revocations and deletions should propagate faster than ordinary content changes.

Should permissions be checked at ingestion or at query time?

Usually both. Copy permissions into the index for fast filtering, and verify fine-grained or recently changed permissions against the source when results are used. Pure query-time checking is accurate but slower; pure ingestion-time copying is fast but only as current as the last sync.

How do we handle deleted documents?

Consume delete events from the source where available, remove all of a document's chunks, and run a periodic reconciliation that deletes any chunk whose document no longer exists or is no longer shared.

What happens when we change embedding models?

Build a new index with the new model, populate it fully, evaluate retrieval quality against the old index, and switch traffic over. Vectors from different models cannot be mixed in one index.

Can we tell the model to ignore content the user may not see?

No. Prompts are not access control. Anything in the context window can end up in the answer, and prompt injection can override instructions. Restricted content must be filtered out before it reaches the model.