RAG in Production: The Failure Modes Nobody Talks About
Retrieval-augmented generation looks trivial in a notebook. In production it fails in five specific ways — here's how we design around each.
Chunking is the silent killer
80% of RAG accuracy problems trace back to how documents were split. Fixed-size chunking destroys semantic boundaries. We use hierarchical chunking with parent-document retrieval — small chunks for search, parent chunks for context — and it consistently outperforms naive splitting.
Stale indexes and drift
A RAG system that was 92% accurate at launch degrades to 60% within a quarter without index hygiene. We ship every RAG deployment with an ingestion pipeline that tracks source-document lineage, freshness, and change detection — and an eval harness that runs nightly against a labeled golden set.
The 'confident wrong' problem
Models hallucinate most confidently when retrieval returns nothing useful. Our guardrail: if retrieval scores fall below a calibrated threshold, the system refuses to answer and routes to a human. This one pattern eliminates the majority of production embarrassment.
Key takeaways
- Invest in chunking strategy before model choice
- Ship an eval harness on day one
- Refuse-to-answer beats confident hallucination
T7 Research
Enterprise AI Research Group
T7 Research is the research arm of T7 Solution, focused on benchmarking LLMs, evaluating RAG patterns, and compiling implementation playbooks for enterprise technology leaders.