How RAG fails in production, and how to fix it

A retrieval-augmented generation demo is easy to build and easy to trust, which is the problem. The version you stand up in an afternoon answers your test questions correctly, so you ship it, and then it degrades in production in ways that are hard to see and almost never the model’s fault. This piece is about that gap: the four places RAG breaks once real documents and real traffic arrive, and the specific fix for each.

It assumes you already know the mechanism. If chunk, embed, retrieve, augment, and generate are not yet familiar, start with what RAG really is; here we skip the assembly and go straight to where the assembled thing fails.

Why the demo hides the failures

A demo asks a handful of questions whose answers sit in obvious places, against a corpus small enough that retrieval can hardly miss. Production asks questions you never tried, against a corpus that grows and changes under you. The failures were there on day one; you were just not measuring them, so the demo looked clean.

There are four of them, and they live at different stages of the pipeline. Three happen before the model ever runs. The fourth is the model misusing context that was handed to it correctly. Each has a known fix, and the diagram below is the whole article in one frame.

Chunking decides the ceiling

How you split documents at index time sets a hard limit on everything downstream, and a fixed-size splitter is the usual culprit. Cut every 500 tokens and you will eventually slice a clause from the condition that governs it, a figure from its unit, or a table row from its header. Once a passage is split wrong, no retriever can fetch the missing half and no model can reconstruct it. The research cataloguing RAG failure points flags chunking and chunk consolidation as failures that occur before the model is ever involved.

The fix is to stop splitting on a token count and start splitting on structure: headings, paragraphs, list items, table boundaries. Add a little overlap so a boundary is a seam rather than a hard cut, and attach metadata (source, section, date) so retrieval can filter. When the best-retrieving unit is small but the model needs more around it, retrieve the small chunk and feed its parent section. For corpora where answers span many passages, hierarchical indexing helps: RAPTOR builds a tree of recursive summaries and shows significant improvements over flat-chunk retrieval, because a summary node can answer a broad question that no single leaf chunk covers.

Retrieval has to work on the whole corpus, not the demo

Dense embeddings capture meaning, which is exactly why they miss exact terms. A query for error code E-4021 or a specific part number can rank a semantically similar passage above the literal match, because the embedding cares about what the text means, not the characters it contains. Dense retrieval beats keyword search by 9 to 19 points on top-20 accuracy on average, and the average is real, but it hides the queries where a keyword match is the only right answer. Then the corpus grows, near-duplicates multiply, and the passage you need slips out of the top results.

The fix is three moves, in order. Run hybrid search: combine the dense similarity score with a sparse keyword signal like BM25, so meaning and exact terms both count. Then rerank: a cross-encoder re-scores the top candidates against the query and reorders them, and BERT-based reranking improved passage ranking by 27 percent relative MRR on MS MARCO. And rewrite the query before you search: generating a hypothetical answer and embedding that (the HyDE method) measurably improves zero-shot retrieval, because a hypothetical answer sits closer in embedding space to the real passage than the bare question does. Embeddings still do the heavy lifting (the embeddings explainer covers why); in production they are the first stage, not the only one.

When the model fights the context

Retrieval can hand over the right passage and the model can still get it wrong. Models carry a confirmation bias toward their own training: they discount retrieved evidence that contradicts what they already believe, unless that evidence is coherent and pointed enough to override it. And the way you pack the prompt makes it worse. Stuff in twenty chunks and the decisive one may land in the middle, where the lost-in-the-middle effect shows models read least reliably; performance is highest at the start and end of the context and sags in between. Reaching for a bigger window is not the escape it looks like, because usable context is shorter than the advertised length and accuracy drops as you fill it.

So the fix runs against the instinct to send more. Send fewer, stronger chunks, and use the reranker to place the best evidence at the start or end rather than the middle. Instruct the model to answer only from the supplied passages, to cite the chunk behind each claim, and to say it does not know when the passages do not support an answer. Then check that it actually did: faithfulness, whether the answer is grounded in the retrieved text, is a measurable property that frameworks like RAGAS score without needing a reference answer.

Retrieval can hand the model the right passage and the model can still answer from its own memory. Grounding is something you instruct and then measure, never something you assume.

The index ages out

The index is a snapshot taken at ingest time, and the world it describes keeps moving. A document gets revised and the old chunk keeps surfacing, so the model answers from a policy that changed last quarter. A document is deleted and its chunks linger. New documents arrive and stay invisible until something indexes them. The survey of the field treats outdated knowledge and continuous updates as a core operational concern, not a setup step you do once.

The fix is to treat indexing as a running pipeline rather than a one-time load. Re-embed on change, ideally triggered by the source system’s own update events, version chunks so you can tell which revision produced an answer, expire stale entries, and monitor the lag between when a source changes and when the index reflects it. A freshness check is far cheaper than a confident, well-cited answer about a policy that no longer exists.

You cannot fix what you do not measure

The reason all four failures stay hidden is that a fluent, cited, wrong answer looks exactly like a fluent, cited, right one. The failure-point research reaches an uncomfortable conclusion on this: a RAG system can only be validated in operation, not reasoned about in advance. Which means the real deliverable is not the pipeline. It is the evaluation set.

Build a fixed set of questions with known-good answers drawn from your actual corpus, and measure the two halves separately. Retrieval: did the correct passage make the top results, scored by recall and rank. Generation: is the answer faithful to what was retrieved, and does it answer the question that was asked. Reference-free frameworks like RAGAS score faithfulness and relevance without a hand-labeled answer for every case, so you can run the whole suite on every index change, every prompt edit, and every model swap, in CI, the way you run tests. Skip it and every fix above is a guess you cannot check.

Where the fixes lead: agentic retrieval

Everything so far treats retrieval as one fixed pass: the same fetch fires for every question. The next step is to let the model run the loop, deciding whether it needs to retrieve at all, judging whether what came back is sufficient, rewriting the query, and searching again before it commits to an answer. Self-RAG trains a model to retrieve on demand and critique its own output, and reports gains in factuality and citation accuracy over a fixed retrieve-then-read setup. At that point retrieval is a tool the model chooses to call, which is the bridge to agents and to the Model Context Protocol that standardizes how a model reaches tools and data in the first place.

What you are actually shipping

A RAG pipeline is a few hundred lines of glue, and almost all of it is replaceable. You will change the embedding model, swap the chunker, add a reranker, move vector stores. The one part that does not change, the part that tells you whether any of those swaps helped or hurt, is the evaluation set. That is the asset. Teams that treat RAG as a one-time integration ship the demo and inherit the slow decay that follows. Teams that treat it as a measured system build the evaluation set first and let it catch the regressions the demo would have waved through. The pipeline produces the answer. The evaluation set is the only thing that tells you the answer was right.