Why enterprise RAG fails quietly, and what to design instead

A retrieval-augmented system passes its demo. Months later it quotes a policy that was replaced last spring, puts a paragraph from an HR-only memo in front of someone who was never cleared to read it, and nobody can say whether it is getting better or worse. Nothing throws an error. The dashboards stay green. That is how enterprise RAG usually fails: quietly, and on the questions that matter most.

We first published this essay in May 2026. For this revision we went back through the research from 2024 to 2026 on long-context models, hybrid retrieval, graph and iterative retrieval, permission-aware search, time-sensitive retrieval and evaluation. The argument held. Several details got sharper, and a few changed how we would build these systems.

Demo corpus, production corpus

The demo corpus is curated. It holds one version of each document, no access rules, and questions written by the people who built the system. The production corpus holds fourteen versions of the same remote-work policy, a memo that only HR may open, form numbers and clause references that embedding models treat as noise, and questions from people who do not know which document they need.

Barnett and colleagues studied three RAG deployments, in research, education and biomedicine, and catalogued seven points where such systems fail. They run from content that is missing altogether, through relevant documents that never reach the top of the ranking, to answers that leave out information the context contained.^[1] Their conclusions for engineers are the part to pin on the wall: a RAG system can only be validated while it operates, and it becomes dependable through operation rather than by design at the start.^[1] Both point the same way. The work that decides whether enterprise RAG succeeds is mostly not model work.

Did long context make retrieval obsolete?

The strongest objection to the first version of this essay was simple. Context windows now hold hundreds of thousands of tokens, some a million. Why retrieve at all? Put the documents in the prompt.

The evidence says this works better than it used to, and less well than the advertised window suggests. A study by Google DeepMind and University of Michigan researchers compared the two approaches on nine long-document datasets with Gemini 1.5 Pro, GPT-4o and GPT-3.5 Turbo. With enough budget, long context scored higher on average. Yet for 63% of queries both approaches gave exactly the same prediction. Their Self-Route method first asks the model whether the retrieved chunks answer the question and sends only the rest to the full context; it matched long-context quality while cutting cost by 65% for Gemini 1.5 Pro and 39% for GPT-4o.^[2] Anthropic’s guidance for its own models draws a similar line from the vendor side: below roughly 200,000 tokens, about 500 pages, include the whole knowledge base in the prompt and use prompt caching; retrieval is for larger corpora.^[3]

The advertised window and the usable window are different numbers. RULER, from NVIDIA researchers, tested 17 long-context models. Nearly all were close to perfect on the plain needle-in-a-haystack test, yet only half of those claiming 32,000 tokens or more held satisfactory performance at 32,000.^[4] NoLiMa removed the literal word overlap between the question and the passage that answers it, which is the normal case in enterprise search, where people rarely use a document’s own phrasing. At 32,000 tokens, 11 of 13 models fell below half of their short-context scores, and GPT-4o dropped from 99.3% to 69.7%.^[5] Models released since may score higher on both benchmarks. What the two papers show is the shape: accuracy falls as the context grows, and it falls further when the wording does not match.

Neither study addresses the point that matters most in an enterprise. Long context changes how much a model can read. It does nothing about what the model is allowed to read, or which version is in force. Put the whole policy library in the prompt and you have put all fourteen versions in the prompt, together with every restricted memo. The filtering problem moves. It does not go away.

So the updated position is this. Use long context to read the right documents in depth, and use retrieval to decide which documents are right: current, permitted and relevant. For a small corpus with a single audience, skip retrieval. For anything with access rules or a version history, the retrieval layer is where those rules live.

Four failures, revisited

The four structural failures from the first version still hold. The research since then puts evidence behind each of them.

1. The metadata that decides correctness is ignored

Vector similarity is one signal. Version, effective date, superseded-by, jurisdiction, classification, owner and locale are others. In many systems only the first is wired into ranking, and the rest sit unused in a metadata column or are never extracted from the source system at all.

Two benchmarks now measure what this costs. HoH, built from evolving real-world knowledge with 96,124 question and answer pairs and 219,463 documents, found that outdated documents substantially reduce answer accuracy by distracting models from the current information, and can mislead them into harmful answers even when the current document is available. The authors conclude that current RAG methods struggle with outdated information in both retrieval and generation.^[6] TimelyRAG, a preprint from September 2026, targets the enterprise case directly: laws, policies and regulations in which amendments override earlier clauses while keeping most of the text. That overlap is the trap. Two consecutive versions of a policy look almost identical to an embedding model, so similarity cannot pick the one in force. Scoring clause-level effective intervals alongside similarity improved nDCG@10 by up to 28.6% in the authors’ experiments.^[7]

The fix is plain. Store valid_from and valid_to on every chunk, per clause where amendments are partial. Take the as-of date from the question when it names one (“what was the rule last March?”) and from the calendar when it does not. Filter on it inside retrieval. Treat a document with no effective date as a defect to fix at ingestion, because no filter can recover dates nobody recorded.

2. Permissions are checked after retrieval

The common pattern retrieves the best chunks from everything, generates an answer, then hides citations the user cannot open. By then the model has read the restricted text. The answer can paraphrase it, a follow-up question can draw it out, and two people asking the same question get different answers for reasons nobody can reconstruct. A permission check after generation is a redaction step, not access control.

OWASP’s Top 10 for LLM applications gives this its own entry in the 2025 edition: LLM08, vector and embedding weaknesses. It names misaligned access controls on embeddings and leaks between user groups that share one vector store, and its first prevention item is fine-grained access control through permission-aware vector stores, with strict logical and access partitioning of datasets.^[8]

Doing this correctly has a side effect to plan for. Microsoft’s documentation for Microsoft 365 Copilot states that Copilot can only summarize or reference content the user is authorized to access, then spends much of the page on controls against oversharing: search restrictions, sharing and membership rules, lifecycle policies and sensitivity labels.^[9] Correct permission trimming faithfully surfaces every folder that was shared too widely years ago. Before an assistant goes live, the access model it inherits needs its own review.

3. Retrieval is only semantic

Dense embeddings handle paraphrase well and exact tokens badly: form numbers, error codes, clause references, product codes, people’s names. BEIR, the standard zero-shot retrieval benchmark, found BM25 to be a strong baseline across domains, with rerankers and late-interaction models (the ColBERT family, which compares query and document token by token) scoring best on average, at a higher compute cost.^[10] Anthropic reported that on its own test sets, combining contextualized embeddings with BM25 cut top-20 retrieval failures by 49% against plain embeddings, and adding a reranker took the reduction to 67%.^[3] Those are vendor numbers on vendor data. The direction matches the public benchmark, and it matches the second of Barnett’s failure points: relevant documents that never make the top of the ranking.^[1]

4. No boundary, no abstention

A system with no stated scope attempts every question. When the retrieved passages do not contain the answer, strong models tend to answer anyway. Google researchers who formalized “sufficient context” found that Gemini 1.5 Pro, GPT-4o and Claude 3.5 answer well when the context is sufficient, but often give incorrect answers instead of abstaining when it is not. Using a sufficiency signal to decide when to answer raised the share of correct answers among those the model gave by 2 to 10%.^[11] Declining is a behaviour you design and then measure, like any other.

Structured search with a semantic ranker

Enterprise retrieval is a structured search problem with a semantic component. The filters that decide correctness (who is asking, what date the question is about, which versions are in force) run inside the retrieval query, so every later stage sees only admissible candidates. Lexical and dense retrieval run side by side over that set, their rankings are fused, and a cross-encoder reranks the short list. The simulator below runs one question through that pipeline. Turn the fixes on one at a time.

One implementation detail catches teams that do the right thing. In pgvector, an approximate HNSW index returns nearest neighbours first and the WHERE clause is applied afterwards. The documentation’s own example: if a filter matches 10% of rows, with the default ef_search of 40, only about 4 rows survive on average.^[12] The permission filter is still enforced, so nothing leaks, but recall collapses without any error for users whose permitted slice of the corpus is small. Since version 0.8.0, iterative index scans keep scanning until enough rows pass the filter, up to a configurable limit; partial indexes and partitioning are the other options.^[12]

-- One row per chunk. ACL and validity are copied from the source system
-- and refreshed whenever sharing or group membership changes.
CREATE TABLE chunks (
  id             bigserial    PRIMARY KEY,
  doc_id         text         NOT NULL,
  version        int          NOT NULL,
  body           text         NOT NULL,
  embedding      vector(1024) NOT NULL,
  tsv            tsvector GENERATED ALWAYS AS (to_tsvector('english', body)) STORED,
  allowed_groups text[]       NOT NULL,  -- who may read the source document
  valid_from     date         NOT NULL,  -- in force from (inclusive)
  valid_to       date                    -- until (exclusive); NULL = still in force
);
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunks USING gin (tsv);
CREATE INDEX ON chunks USING gin (allowed_groups);

-- Per request, inside a transaction.
-- $1 query embedding, $2 query text, $3 the caller's groups from the identity
-- provider (never from the prompt), $4 the date the question is about.
SET LOCAL hnsw.iterative_scan = strict_order;  -- pgvector 0.8+: scan past filtered-out rows

WITH dense AS (
  SELECT id, rank() OVER (ORDER BY embedding <=> $1) AS r
  FROM chunks
  WHERE allowed_groups && $3
    AND valid_from <= $4 AND (valid_to IS NULL OR valid_to > $4)
  ORDER BY embedding <=> $1
  LIMIT 40
),
lexical AS (
  SELECT id, rank() OVER (ORDER BY ts_rank_cd(tsv, q) DESC) AS r
  FROM chunks, websearch_to_tsquery('english', $2) AS q
  WHERE tsv @@ q
    AND allowed_groups && $3
    AND valid_from <= $4 AND (valid_to IS NULL OR valid_to > $4)
  ORDER BY ts_rank_cd(tsv, q) DESC
  LIMIT 40
)
SELECT c.id, c.doc_id, c.version, c.body,
       coalesce(1.0 / (60 + d.r), 0) + coalesce(1.0 / (60 + l.r), 0) AS rrf
FROM dense d
FULL OUTER JOIN lexical l ON l.id = d.id
JOIN chunks c ON c.id = coalesce(d.id, l.id)
ORDER BY rrf DESC
LIMIT 20;  -- a cross-encoder reranks these; the top few go to the model

Three things about this query matter more than its syntax. The group list comes from the identity provider for the signed-in user, never from text in the prompt. The same admissibility predicate appears in both branches, so neither ranker can reach around it. And the date is a parameter, so a question about last March runs against last March’s rules. Postgres row-level security can enforce the same predicate a second time inside the database, so a new code path that forgets the WHERE clause still sees only what the user may see. Connect as a role that does not own the table, because table owners bypass row-level security by default.

Permissions are an identity problem

The boundary is the identity of the person at the moment they ask. Retrieval runs under that identity, and generation runs on context that has already been filtered. The model never sees what the user could not open themselves.

Where source systems can answer “may this user read this document?” at query time, ask them. Most cannot do it fast enough for every candidate, so the practical design is a permission projection: ACL columns copied onto every chunk and refreshed when group membership or sharing changes. The refresh lag is then a security property. Measure it, alert on it, and treat a stale projection like any other access-control incident. Where tenants or clearance bands must never mix, give them separate indexes or partitions, as OWASP recommends,^[8] rather than trusting one filter to hold forever.

When the question is not a lookup

Two kinds of question defeat single-shot top-k retrieval, and both have useful research behind them.

The first is the global question: what are the recurring themes in two years of incident reports? No single chunk answers it. Microsoft Research’s GraphRAG uses a language model to build an entity graph from the corpus, pre-generates summaries for communities of related entities, and answers global questions by combining those summaries. The paper shows conventional vector RAG failing on this class of question.^[13] The price is an indexing pass in which a model reads the whole corpus. In an enterprise that index needs the same boundaries as everything else: a community summary built from restricted documents is itself restricted, and one built from superseded versions is itself stale.

The second is the multi-step question, where the answer to one lookup decides the next. The long-context study above names this among the main reasons retrieval lost: the results of earlier steps are needed to retrieve for later ones, and general or implicit questions give the retriever little to search with.^[2] Iterative, or agentic, retrieval lets the model issue follow-up searches. The engineering rule is that every search the agent makes goes through the same identity-bound retrieve function. An agent holding a raw database connection is a permission bypass with a friendly interface.

Evaluation that measures the parts

We still treat evaluation as the load-bearing part of a production RAG system. What changed is our view of what it has to measure.

An end-to-end answer score hides the failures above. A system can score well while leaking on every question that touches restricted content, because those questions are a small slice of the set and the leaked answers are often correct. Fixing the leak can even lower the score, because the honest answer to some of those questions is that no document this person may read answers them. The scorecard below shows the effect on a toy test set.

So the suite measures stages separately and reports by slice:

• Retrieval. Precision and recall against labelled relevant chunks, plus must-not-retrieve labels for superseded and restricted chunks. A restricted chunk in the context is a failure even when the answer is right.
• Sufficiency and abstention. For questions the corpus cannot answer, did the system decline? In the sufficient-context work, Gemini 1.5 Pro labelled sufficiency with 93% accuracy without needing a reference answer, which makes the check cheap to run at scale.^[11]
• Groundedness. Does each claim in the answer trace to a retrieved passage?
• Completeness. The TREC 2024 RAG Track revived nugget evaluation, which scores an answer by the atomic facts it contains. A fully automatic version built on language models correlated strongly with mostly manual scoring by human assessors across 21 topics and 45 runs.^[14]

Language-model judges make this affordable, and they need calibration. ARES scores context relevance, answer faithfulness and answer relevance with small fine-tuned judges, then uses a few hundred human-labelled examples to correct the judges’ errors statistically, a method called prediction-powered inference.^[15] That is the right shape for an enterprise suite: a small labelled set, owned by the people who know the domain and versioned with the code, keeping a larger automated suite honest.

And because a RAG system can only be validated in operation,^[1] the suite runs on sampled production questions on a schedule, not only in CI before a release. Corpora change weekly. Permissions change daily. A test set frozen at launch measures a system that no longer exists.

from collections import defaultdict
from dataclasses import dataclass, field
from typing import Callable, Optional


@dataclass
class Case:
    slice: str        # "lookup", "exact-id", "version", "restricted", "unanswerable"
    user: str
    question: str
    as_of: str        # ISO date the question is about
    relevant: set[str]                                   # chunks a full answer needs; empty if unanswerable
    restricted: set[str] = field(default_factory=set)    # chunks this user must never be shown
    superseded: set[str] = field(default_factory=set)    # older versions of the relevant chunks


Retrieve = Callable[[str, str, str, int], list[str]]                # (user, question, as_of, k) -> chunk ids
Answer = Callable[[str, list[str]], tuple[Optional[str], list[str]]]  # -> (text or None to decline, claims)
Supported = Callable[[str, list[str]], bool]  # judge: is this claim backed by these chunks?


def evaluate(cases: list[Case], retrieve: Retrieve, answer: Answer, supported: Supported, k: int = 5):
    totals: dict[str, dict[str, float]] = defaultdict(lambda: defaultdict(float))
    for case in cases:
        ids = retrieve(case.user, case.question, case.as_of, k)
        text, claims = answer(case.question, ids)
        s = totals[case.slice]
        s["cases"] += 1
        s["leaks"] += bool(case.restricted & set(ids))   # a leak even if the answer is right
        s["stale"] += bool(case.superseded & set(ids))
        if case.relevant:
            hits = len(case.relevant & set(ids))
            s["answerable"] += 1
            s["recall"] += hits / len(case.relevant)
            s["precision"] += hits / max(len(ids), 1)
            if text is not None:
                s["answered"] += 1
                s["grounded"] += bool(claims) and all(supported(c, ids) for c in claims)
        else:
            s["unanswerable"] += 1
            s["declined"] += text is None

    def ratio(a: float, b: float) -> Optional[float]:
        return round(a / b, 3) if b else None

    return {
        name: {
            "cases": int(s["cases"]),
            "leak_rate": ratio(s["leaks"], s["cases"]),
            "stale_rate": ratio(s["stale"], s["cases"]),
            "recall": ratio(s["recall"], s["answerable"]),
            "precision": ratio(s["precision"], s["answerable"]),
            "grounded": ratio(s["grounded"], s["answered"]),
            "correct_declines": ratio(s["declined"], s["unanswerable"]),
        }
        for name, s in totals.items()
    }


def release_gate(report: dict, min_recall: float = 0.9, min_grounded: float = 0.95) -> list[str]:
    """Block the release if any slice leaks or goes stale, whatever the averages say."""
    problems = []
    for name, m in report.items():
        if m["leak_rate"]:
            problems.append(f"{name}: restricted chunks retrieved in {m['leak_rate']:.0%} of cases")
        if m["stale_rate"]:
            problems.append(f"{name}: superseded chunks retrieved in {m['stale_rate']:.0%} of cases")
        if m["recall"] is not None and m["recall"] < min_recall:
            problems.append(f"{name}: recall {m['recall']:.2f} is below {min_recall}")
        if m["grounded"] is not None and m["grounded"] < min_grounded:
            problems.append(f"{name}: grounded {m['grounded']:.2f} is below {min_grounded}")
    return problems

The harness does not need to be clever. It needs must-not-retrieve labels, slices that match your real risks, and a gate that fails the release when any slice leaks, however good the average looks. The supported judge is where a calibrated language-model judge plugs in.

What to design instead

The systems that hold up in production share a short list of properties. None of them is a choice of model.

1. Admissibility inside retrieval. Identity and the as-of date are query parameters. Rankers only see chunks the user may read that are in force on the date asked about.
2. Hybrid retrieval with reranking. Lexical and dense retrieval side by side, fused, then reranked. Exact identifiers get their own test slice.
3. Versions as data. Effective dates and supersession recorded per document, and per clause where amendments are partial. Missing dates are ingestion defects.
4. Long context for depth, not for access. Read whole admitted documents when the question needs them. For a small corpus with a single audience, skip retrieval.
5. Explicit scope and measured abstention. State what the system will not answer, decline cleanly, hand off to people, and count declines in the evaluation.
6. Agents inherit the boundary. Every tool call that fetches context goes through the same identity-bound retrieve function, and derived indexes such as graph summaries carry the permissions and dates of their sources.
7. Evaluation by stage and slice, on live samples. Retrieval, abstention, groundedness and completeness scored separately, calibrated against human labels, and rerun on production traffic.
8. Audit by default. Log the user, their groups, the as-of date, candidate IDs, scores, prompt and model version for every answer, so “why did it say that?” has a reproducible answer.

None of this is exotic, and most of it is not new. What the last two years of research add is evidence: bigger windows do not remove the need to filter, similarity cannot tell versions apart, and averages hide the failures that matter. Most enterprise RAG fails because it was built to demo rather than to operate. The fixes are the ones production engineering has always asked for, applied to retrieval.

If you have inherited a RAG system that is quietly failing, or are about to build one, this is the work our AI agent engagements (opens in a new tab) cover, usually alongside the data infrastructure (opens in a new tab) the corpus depends on.