Retrieval-augmented generation is the dominant architecture for AI in professional services, because it is the one that grounds answers in a firm's own documents rather than in a model's training. It is also the architecture whose characteristic failure a reviewer is least equipped to notice.
Key Takeaway
Practitioner sources converge on one diagnosis: the failure is almost never in generation, and when a system gives a wrong or incomplete answer the root cause is usually retrieval, having fetched the wrong material or none at all. One source reports that 80 percent of failures trace to the ingestion and chunking layer rather than the model, and describes teams spending weeks tuning prompts and swapping models while retrieval quietly returns the wrong context. A data governance vendor reports the same pipeline achieving 85 to 92 percent accuracy on governed data against 45 to 60 percent on ungoverned data, arguing data quality sets a ceiling that algorithm tuning cannot fully close. Against the near-universal practitioner advice to abandon fixed-size chunking, an academic evaluation finds chunking a much more difficult and fragile problem than is often assumed, with many methods failing in practice through timeouts, memory issues or poor scalability, only a subset including fixed-size completing consistently, and results varying strongly by document type.
The Diagnosis Everyone Gets Wrong
The finding that reorients where a firm should look, and it is the most consistent claim across these sources.
One source states it plainly: the failure is almost never in generation, and when a system gives a wrong, hallucinated or incomplete answer, the root cause is usually retrieval, because the system fetched the wrong chunks or none at all. It adds that the generator is doing its job and the retriever is not giving it the material it needs[1].
Another agrees that most failures start in retrieval rather than generation, since when a system retrieves incomplete, irrelevant or poorly ranked evidence, even a strong model will produce weak answers[2].
A third puts a number on it, stating that 80 percent of failures trace back to the ingestion and chunking layer rather than the model, and observing that most teams discover this after spending weeks tuning prompts and swapping models while their retrieval quietly returns the wrong context every third query[3].
We report these proportions as stated; all three sources are commercial and none carries a citation for its figure.
The behavioural observation is the useful part regardless of the number. Prompts and model choice are the visible, adjustable parts of the system, so that is where attention goes. Retrieval is upstream, produces no error, and is not displayed to anyone.
Why This Failure Is Invisible
The mechanism that makes retrieval failure worse than generation failure for a professional firm. This section is our own analysis, building on findings earlier in this series.
The hallucination measurement article distinguished faithfulness, meaning whether an output is supported by the supplied context, from correctness, meaning whether it is true. The two come apart precisely here.
A retrieval failure supplies the wrong document. The model then does exactly what it should: it produces an answer faithfully grounded in what it was given. The output is coherent, sourced, internally consistent and wrong.
Every quality signal a firm is likely to check will pass. A faithfulness score compares the answer to the retrieved context and finds agreement. A reviewer checking the answer against the cited passage finds agreement. A citation appears and resolves to a real document.
The only check that catches it is asking whether the retrieved document was the right one, which requires knowing what should have been retrieved, which is the reviewer redoing the search.
This is why the incident response article identified retrieval context as the decisive artefact for grounded workflows, and why it stated that an output can be perfectly faithful to a wrong document with the failure invisible unless the retrieval is recorded.
The practical instruction follows directly: a review that checks the answer against the sources supplied is not checking retrieval. It is confirming the model behaved, which was never the likely failure.
The Layer Everyone Skips
Where the problem starts, and the section a Canadian professional firm should read most carefully.
One source is emphatic: most guides skip this stage, and it is where production retrieval actually starts failing. Raw documents are not clean text. PDFs have tables, headers, footers, multi-column layouts and scanned pages. HTML has navigation menus mixed into the body. Word documents have tracked changes and comments. If the parser returns garbage, the chunks are garbage, the embeddings are garbage and retrieval returns garbage, regardless of how well everything downstream is configured[3].
It recommends layout-aware parsing that distinguishes body text from headers, tables and figures, rather than basic text extraction libraries, and an optical character recognition stage for scanned documents[3].
Read that list against the document population of a Canadian accounting or advisory practice. Scanned supplier invoices. Financial statements whose meaning is carried entirely by table structure. Agreements in multi-column layout with defined terms. Documents with tracked changes that were never accepted. Bank statements as images.
Almost every characteristic named as a parsing hazard describes the ordinary working material of a professional firm, and the hazard is more acute than for the general case because a table in a financial statement is not decoration. Losing the column structure loses the meaning entirely.
This also connects to the prompt injection article, which argued that the rendered document and the extracted text stream are different objects. Here the same divergence produces a quality failure rather than a security one, and the same remedy applies: look at what the parser actually produced.
Data Quality Sets The Ceiling
The structural argument, reported with a clear note about who is making it.
One source reports that the same query through the same pipeline produces 85 to 92 percent accuracy on governed data and 45 to 60 percent on ungoverned data, describing that 30 to 45 percentage point gap as a data governance infrastructure problem that retrieval algorithm tuning cannot fully close. It states that most engineering focuses on chunking, embedding model selection and reranking, that these fixes are real and that hybrid retrieval with reranking delivers documented accuracy improvements, but that they are bounded by the quality of the data feeding the pipeline, so data quality determines the ceiling[4].
That source is a data governance vendor, and the finding it reports is that organisations need data governance. We flag that plainly and note the research cited is its own.
The structural claim nonetheless coheres with material this publication has reported from unrelated directions. The total cost article found data preparation reported as the largest single cost category in AI implementation, at 20 to 50 percent of project budgets depending on the source. This offers an explanation for why: the data work is not preparatory overhead but the determinant of the achievable outcome.
For a Canadian firm the translation is a sequencing rule. If retrieval accuracy is unsatisfactory, the first question is whether the underlying documents are organised, current and unambiguous, because tuning against a poor corpus optimises within a ceiling that the corpus set.
The recognisable version in a professional practice is a document store with multiple versions of the same agreement, superseded policies never removed, duplicates across folders, and no reliable indication of which version is current. Retrieval will find all of them and cannot know which one you meant.
Similarity Is Not Relevance
A conceptual error at the heart of naive implementations, illustrated well by one source.
It gives the example that "How do I reset my password?" and "Password reset policy" have high similarity but serve different intents, and lists the classic pipeline's predictable failure modes as chunking destroying context, embedding similarity not equalling relevance, top-K retrieval being crude, and no query understanding, since the raw user query goes straight to vector search with no transformation. Its summary is that naive retrieval, meaning chunk, embed, cosine similarity and stuff into the prompt, was always a prototype rather than a production system[5].
The distinction between similarity and relevance is worth translating into professional terms, and this is our own analysis.
A query about whether a particular expense is deductible is textually similar to a passage explaining that a similar expense is not deductible. The two passages sit close together in any embedding space, because they concern the same subject in nearly the same words.
Similarity measures aboutness. Professional work turns on distinctions within a subject, frequently on a single qualifying condition. A retrieval mechanism that ranks by aboutness is indifferent to precisely the distinction that matters.
That is a structural reason to expect retrieval to perform worse on technical professional questions than on general ones, independent of implementation quality, and it argues for metadata filtering and for retrieval designs that use the structure of documents rather than only their text.
Top-K Is Crude
The retrieval mechanic itself, and the standard remedy.
One source notes that the five most similar chunks are not necessarily the five most useful[5]. Another reports that hybrid search combining keyword and vector retrieval with score fusion, together with a cross-encoder reranker, reduces error rates substantially against naive vector-only retrieval[1], while a third reports a smaller improvement figure for hybrid retrieval with reranking[4].
We note that the two reported improvement figures for what is broadly the same intervention differ considerably, and that both are unsourced.
The architectural pattern the sources agree on is more useful than either number. One describes it as retrieving a large pool, reranking aggressively, and passing only a small final context to the model, characterising this as the standard production architecture with the reranker doing the heavy lifting so the model sees only the highest-quality evidence[6].
The reason this matters for a firm buying rather than building is that it is a checkable question. Does the system rerank, or does it pass the top few vector matches directly to the model? The second is the prototype pattern, and a vendor should be able to answer.
Context Saturation
A failure mode with a counterintuitive shape: supplying more evidence makes the answer worse.
One source works through the arithmetic. Ten chunks at eight hundred tokens each is eight thousand tokens of context before the query and system prompt, which might add another two thousand, reaching ten thousand tokens, at a range where instruction adherence starts degrading. Adding conversation history for a multi-turn application saturates the model's effective reasoning window[6].
It lists context window saturation among four structural reasons systems fail, describing too many retrieved chunks as degrading instruction adherence[6].
We report the specific token thresholds as stated and note they are model-dependent and will change.
The general finding matters because it inverts an intuition that firms act on. Faced with retrieval that misses relevant material, the obvious response is to retrieve more, and the sources indicate that beyond a point this degrades the answer rather than improving it.
It also has a cost consequence the inference economics article would predict. Retrieved context is input tokens on every request, so a firm that responded to quality concerns by widening retrieval has raised its cost per task and may have lowered its answer quality simultaneously.
The Chunking Consensus
The advice that appears in almost every practitioner source, stated before the section that complicates it.
One source is unequivocal: fixed-size chunking is the root cause of retrieval failures in most pipelines, and a firm should switch to semantic or proposition-based chunking first because it costs almost nothing and lifts retrieval accuracy dramatically. It adds that chunking strategy constrains retrieval accuracy more than embedding model choice, and reports a 2025 clinical study in which adaptive chunking achieved 87 percent retrieval accuracy against 13 percent for fixed-size baselines on the same dataset[1].
Another lists chunking destroying context as the first failure mode, noting that splitting at a fixed token count breaks paragraphs, separates questions from answers and loses document structure[5]. A third names chunk boundaries splitting semantic meaning as its first of four structural reasons, stating that fixed-size chunking on mixed document types destroys retrieval quality at scale[6].
That is a strong consensus with a clear instruction, and the reported 87 against 13 percent gap is dramatic enough that a reader would reasonably act on it.
We report it at second hand, having not accessed the clinical study, and note that a gap of that magnitude on a single dataset in a single domain is unusual and would warrant checking before being generalised.
The Academic Corrective
The finding that complicates the consensus, and the reason this article exists in the form it does.
An academic evaluation of chunking methods, assessing effectiveness against computational cost and limitations, concludes that chunking in retrieval systems is a much more difficult and fragile problem than is often assumed. It reports that many methods fail in practice due to timeouts, memory issues or poor scalability, and that only a subset, which it names as recursive semantic, fixed-size, sequential hierarchical clustering and max-min, consistently completes processing across datasets. It further states that results vary strongly depending on document type and implementation constraints, that execution time differences between methods are substantial, and that chunking performance cannot be assessed without considering robustness and efficiency rather than retrieval metrics alone[7].
Three of those findings sit awkwardly beside the practitioner advice above.
Fixed-size chunking appears among the methods that consistently complete, rather than as the method to abandon.
The claim that switching costs almost nothing is difficult to reconcile with substantial execution time differences and with methods failing on timeouts and memory.
And results varying strongly by document type means that a result obtained on one corpus, including the clinical dataset cited above, does not transfer to another.
The paper's own summary is the sentence we would give most weight: chunking is a much more difficult and fragile problem than is often assumed[7].
Reading The Conflict
How a Canadian firm should hold two sets of sources that disagree. This section is our own analysis.
We are not in a position to adjudicate, and we would not on the strength of one evaluation against several practitioner accounts. What we can do is note that the two are answering different questions.
The practitioner sources ask which method retrieves best when it works. The evaluation asks which methods complete reliably, at what computational cost, across document types. Those are different criteria, and a method can win the first while failing the second.
The reconciliation we would offer is that advanced chunking may well outperform where it runs successfully on the corpus in question, and that whether it does either is a property of your documents rather than a general fact.
Which produces the only defensible instruction: test chunking strategies on your own corpus and measure both retrieval quality and whether the process completes at your scale. The evaluation's central point is that the second question is real and is usually not asked.
This is also, once again, the private evaluation set. A firm with a set of representative questions and known-correct source documents can measure any chunking change directly. A firm without one is choosing between competing claims about someone else's corpus.
Embedding Drift
A failure mode that connects this article to two others in the series.
One source lists embedding model drift among its four structural failures, stating that switching embedding models without re-indexing corrupts vector similarity scores, and describes the primary mechanism as the model used at index time differing from the model at retrieval time[6].
The failure is silent and total. Vectors produced by two different models occupy different spaces, so comparing them yields numbers that look like similarity scores and mean nothing.
Two connections matter for a Canadian firm.
The drift article identified semantic drift as a mode in which the geometry of the retrieval space shifts, with queries surfacing different documents, no error and no alert. This is the sharpest version of that: a change nobody classified as a system change silently invalidates every stored vector.
And the lock-in article identified embeddings as a dependency requiring index rebuilds on any migration, with the durable principle that source documents are the asset and vectors are derived. A firm that retains documents, chunking rules and pipeline configuration can rebuild after any embedding change, paying compute. A firm that cannot has a corpus it can neither migrate nor repair.
The Fallback That Is Wrong For Finance
A widely recommended design pattern that we think a professional firm should reject, and we say so directly.
One source describes adaptive retrieval as the current standard, in which the system classifies each query, routes to the right retrieval strategy, and falls back to the model's parametric knowledge when retrieval confidence is low[1]. Another recommends implementing a fallback in which, if retrieval confidence is low, the system falls back to a direct model response with a disclaimer[5].
For general applications that is reasonable. For a Canadian professional workflow we think it is the wrong default, and this is our own analysis.
The reason a firm chose retrieval was to ground answers in its own authoritative documents rather than in a model's training. Falling back to parametric knowledge when retrieval fails abandons exactly that property at exactly the moment it was needed, and does so silently from the user's perspective.
Worse, it inverts the failure mode. Retrieval returning nothing is a visible, useful failure: the firm learns the corpus lacks something. Retrieval returning nothing and the model answering from memory produces a confident, ungrounded answer that looks identical to a grounded one.
The disclaimer does not repair this, for the reasons the oversight article established. Explanatory and cautionary text accompanying an output has not been shown to increase scrutiny, and the reported evidence there ran the other way.
The correct fallback for professional work is to decline and route to a person. A system that says it cannot find supporting material is providing accurate and actionable information; a system that answers anyway is not.
The Silent Fade
The trajectory over time, which one source describes precisely.
It observes that queries become ambiguous, the corpus expands, retrieval quality drops, latency balloons and the system silently begins to fade in accuracy, adding that poor evaluation techniques hide where the system actually begins to fail until users complain. Its conclusion is that teams must approach retrieval as a living system rather than a set-and-forget demo architecture[2].
The word silently is the same word the drift article used, and the mechanism is the same one.
The corpus-growth mechanism is worth stating for a professional practice, because it is guaranteed rather than possible. A firm's document store grows every year, and retrieval quality is a function of how many plausible matches exist for a query. A corpus with one relevant document per question retrieves well; the same system with five years of accumulated near-duplicates does not.
So retrieval performance degrades as a direct consequence of the firm operating normally, with no change to any configuration.
That is an argument for the scheduled evaluation this series has recommended repeatedly, and here the reason is unusually clear. There is a known, continuous, unavoidable source of degradation, and no alert accompanies it.
Measuring Retrieval, Not Just Answers
The measurement framework, and the important distinction within it.
One source names five measurable production metrics from a common evaluation framework: faithfulness, answer relevancy, context precision, context recall and answer correctness, suggesting targets above defined thresholds for the first two[1]. Another emphasises measurement of groundedness and faithfulness and continuous evaluation[2].
The distinction a Canadian firm should draw from that list is between the metrics that assess the answer and the metrics that assess retrieval.
Faithfulness and answer relevancy describe the generated answer relative to what it was given. Per the invisible failure argument above, both can be excellent while the system is systematically wrong.
Context precision and context recall are the retrieval metrics, describing whether the right material was fetched. Recall in particular answers the question that matters most: did the system find what it should have found.
A firm monitoring only faithfulness is monitoring the component that was not the likely failure, which is the same error the diagnosis section described, expressed in metrics rather than in engineering effort.
Log What Was Retrieved
The single practice that supports everything above, and a fourth reason to do something this series has already recommended twice.
One source recommends monitoring retrieval quality by logging which chunks were retrieved and whether users found answers helpful[5].
That artefact has now appeared in four separate articles for four separate reasons. The incident response article required it as one of the four forensic artefacts that cannot be captured retroactively. The lineage article identified it as the decisive record for grounded workflows and as the thing that distinguishes a faithful summary of the wrong document from an unfaithful summary of the right one. The prompt injection article argued reviewers should see what the model read. And here it is the only means of diagnosing a retrieval failure at all.
The practical consequence is that a firm which logs retrieval can answer four questions it otherwise cannot: what happened in an incident, what supported a given output, whether a payload was present, and why the answer was wrong.
A firm that does not log retrieval will, when its system produces a wrong answer, be unable to determine whether the material was absent from the corpus, present but not retrieved, retrieved but outranked, or retrieved and misread. Those four causes have entirely different remedies, and without the log they are indistinguishable.
The Canadian Document Problem
Consolidating why this architecture is harder for a professional firm than for a general business. This section is our own analysis.
Four properties of professional document populations make retrieval harder, and all four are ordinary rather than exceptional.
Structure carries meaning. A financial statement's figures are meaningful only in their table position. Parsing that flattens structure destroys the content rather than degrading it.
Versions accumulate and matter. Which version of an agreement, which year's rate, which superseded policy. Similarity-based retrieval cannot distinguish current from historic without metadata, and the historic version is textually almost identical.
Distinctions are narrow. Professional questions frequently turn on a single qualifying condition, and the passage stating the condition and the passage stating the exception are maximally similar.
Scanned material is common. Client-supplied documents arrive as images, requiring an accurate character recognition stage before anything downstream can work.
None of this argues against the architecture. It argues that a Canadian professional firm should expect worse out-of-the-box performance than general benchmarks suggest, should invest disproportionately in ingestion and metadata rather than in model selection, and should be sceptical of accuracy figures obtained on cleaner corpora.
What Order To Fix Things In
A sequence, given that the sources agree on components and not always on priority.
First, look at what the parser produced. If extraction is losing tables, headers or scanned content, nothing downstream can recover it.
Second, address the corpus. Duplicates, superseded versions and missing currency markers set a ceiling that tuning cannot lift.
Third, add metadata filtering. Filtering by date, document type or entity before vector search reduces the population in which similarity has to discriminate.
Fourth, add reranking. Retrieve broadly, rerank, pass a small final context.
Fifth, revisit chunking, and test it on your own corpus. The consensus favours semantic approaches; the academic evaluation counsels checking robustness and cost on your document types.
We would put ingestion and corpus quality first because they bound everything else, and because they are the two items a firm can address without any specialist capability. Reranking and chunking are engineering decisions; document hygiene is a records management decision that a professional practice is already equipped to make.
A Worked Case: Every Third Query
A Canadian advisory firm whose document assistant gives confident wrong answers. The reconstruction illustrates the diagnostic path rather than reporting a specific engagement.
The firm's response is to change the prompt, then to change the model. Neither helps, which is the pattern one source describes as weeks of tuning while retrieval quietly returns the wrong context[3].
It does not help because the generator was never the problem[1]. The failures are faithful answers to wrong documents, so every faithfulness check passes and every reviewer comparing answer to cited source finds agreement.
The actual causes are ordinary. Scanned client documents were parsed without a character recognition stage, so their content is largely absent from the index[3]. Superseded versions of key documents remain in the store and are textually near-identical to current ones[4]. And retrieval passes the top vector matches straight to the model with no reranking[6].
The firm cannot see any of this because retrieval is not logged, so it cannot distinguish material that was absent from material that was present and not retrieved[5].
Two of the three causes are records management problems rather than engineering problems, and the firm was looking at the model.
What To Do
Look upstream first. The failure is almost never in generation, and prompt and model changes address the component that was working.
Inspect what the parser produced. Tables, headers, multi-column layouts and scanned pages are where professional documents break, and structure carries meaning in this material.
Fix the corpus before tuning the algorithm. Duplicates, superseded versions and absent currency markers set a ceiling.
Log retrieval. Without it you cannot tell whether material was absent, unretrieved, outranked or misread, and those have different remedies.
Measure context precision and recall, not only faithfulness. Faithfulness passes on exactly the failure you care about.
Ask whether the system reranks. Passing top vector matches directly to the model is the prototype pattern.
Do not simply retrieve more. Beyond a point, additional context degrades instruction adherence and raises cost per task simultaneously.
Reject the parametric fallback. For professional work the correct response to low retrieval confidence is to decline and escalate, not to answer from training with a disclaimer.
Test chunking on your own documents. The consensus and the academic evaluation disagree, results vary strongly by document type, and robustness matters as much as retrieval scores.
Schedule re-evaluation. Corpus growth degrades retrieval as a consequence of operating normally, with no alert.
The Limits Of This Analysis
Several caveats matter. Most sources here are commercial publications by vendors of data governance, cloud infrastructure, model platforms and consulting, and several report headline proportions, including the 85 percent development-to-production failure claim, the 80 percent ingestion attribution and the roughly 40 percent retrieval failure figure, without citations; we report them as stated and not as established. The governed against ungoverned accuracy comparison is reported by a data governance vendor citing its own research and concluding that data governance is required. The reported clinical chunking result of 87 against 13 percent reaches us at second hand and we did not access the study; a gap of that size on a single dataset warrants verification before generalisation. Two sources give materially different improvement figures for broadly the same hybrid retrieval and reranking intervention. Token thresholds for context degradation are model-dependent and will change. The academic evaluation we rely on for the corrective is an arXiv preprint whose peer review status at the version accessed is unknown to us, and we have reported its conclusions rather than assessing its methodology. The invisible failure argument, the similarity-versus-distinction analysis, the rejection of parametric fallback for professional work, the four properties of Canadian professional document populations, the retrieval logging synthesis and the worked case are our own analysis. This article does not address vector database selection, latency engineering, graph-based retrieval, agentic retrieval patterns, or the security dimension of retrieval, some of which this publication treats separately. Nothing here is a substitute for professional advice on system design.
Frequently Asked Questions
Our answers are wrong. Should we change the model?
Why is this failure so hard to notice?
Should we abandon fixed-size chunking?
What should happen when retrieval finds nothing?
Why is this harder for a professional firm?
Will retrieval quality stay where it is?
References
- Khan, M. (2026, May 16). Production RAG: Why Retrieval Fails and How to Fix It, on the failure almost never being in generation and the root cause usually being retrieval, the reported figure for naive retrieval failing to fetch correct context, fixed-size chunking as the root cause with semantic or proposition-based chunking recommended, the reported clinical study comparing adaptive and fixed-size chunking accuracy, hybrid search with reciprocal rank fusion and a cross-encoder reranker reducing error rates, the five evaluation metrics with suggested targets, and adaptive retrieval with parametric fallback described as the current standard. Note: a commercial technical blog; figures are unsourced and the clinical study was not accessed. mudassirkhan.me/blog/production-rag-guide-2026
- DigitalOcean. (2026, April 29). Why RAG Systems Fail in Production, on most failures starting in retrieval rather than generation, the corpus expanding while retrieval quality drops and the system silently fading in accuracy, poor evaluation hiding failure until users complain, the requirement for retrieval engineering, latency management, drift mitigation and continuous evaluation, and the framing of retrieval as a living system rather than a set-and-forget architecture. Note: published by a cloud infrastructure vendor. digitalocean.com/community/conceptual-articles/why-rag-systems-fail-in-production
- PremAI. (2026, March 17). Building Production RAG: Architecture, Chunking, Evaluation and Monitoring, on the reported attribution of most failures to the ingestion and chunking layer rather than the model, teams tuning prompts and swapping models while retrieval returns the wrong context, raw documents not being clean text with PDFs containing tables, headers, footers, multi-column layouts and scanned pages, the garbage-in consequence for chunks, embeddings and retrieval, and the recommendation of layout-aware parsing with an optical character recognition stage for scanned material. Note: published by a model platform vendor; the headline proportion is unsourced. blog.premai.io/building-production-rag-architecture-chunking-evaluation-monitoring-2026-guide
- Atlan. (2026, May 18). RAG Accuracy Problems: Why RAG Fails and How to Fix It, on the four layers where accuracy breaks down, the reported comparison of the same pipeline achieving 85 to 92 percent accuracy on governed data against 45 to 60 percent on ungoverned data, the argument that data quality determines the ceiling and that retrieval tuning cannot fully close the gap, the reported accuracy improvement from hybrid retrieval with reranking, and the claim that a large majority of systems working in development fail in production. Note: published by a data governance vendor citing its own research and concluding that data governance is required; headline figures are unsourced. atlan.com/know/rag-accuracy-problems
- Gao, Y. (2026, March 21). RAG Is Not Dead: Advanced Retrieval Patterns That Actually Work in 2026, on the classic pipeline's predictable failure modes of chunking destroying context, embedding similarity not equalling relevance with the password example, top-K retrieval being crude and no query understanding; the characterisation of naive retrieval as always having been a prototype; and the recommendations to monitor retrieval quality by logging which chunks were retrieved, use metadata filters before vector search, and implement a low-confidence fallback with a disclaimer. Note: a developer community post. dev.to/young_gao/rag-is-not-dead-advanced-retrieval-patterns
- AI Vanguard. (2026, July 3). RAG in Production: What Nobody Tells You Before You Deploy, on the four structural failure reasons of chunk boundaries splitting semantic meaning, embedding model drift where the model at index time differs from retrieval time, absence of reranking retrieving topically related but contextually wrong documents, and context window saturation degrading instruction adherence; the worked token arithmetic; and the standard production pattern of large retrieval pool, aggressive reranking and small final context. Note: a commercial technical publication; token thresholds are model-dependent. aivanguard.tech/rag-in-production-2026-enterprise-guide
- Chunking Methods on Retrieval-Augmented Generation: Effectiveness Evaluation Against Computational Cost and Limitations. arXiv preprint 2606.00881, on chunking being a much more difficult and fragile problem than often assumed, many methods failing in practice due to timeouts, memory issues or poor scalability, only a subset including fixed-size consistently completing across datasets, results varying strongly by document type and implementation constraints, substantial execution time differences, and the conclusion that chunking performance cannot be assessed without considering robustness and efficiency rather than retrieval metrics alone. Note: a preprint; peer review status at the version accessed is unknown to us and we report its conclusions rather than assessing its methodology. arxiv.org/pdf/2606.00881
This article discusses retrieval system design and is provided for general informational purposes. Most sources are commercial publications by vendors of related products, several headline proportions are reported without citations, and one academic evaluation contradicts the practitioner consensus on chunking, which we report rather than resolve. Nothing here is a substitute for professional advice on system design.