A retrieval demo takes an afternoon. A retrieval system you can trust in production takes discipline. Here's where that discipline actually goes.
Retrieval-augmented generation is the default pattern for grounding language models in your own data, and for good reason: it keeps answers current, cuts hallucination, and avoids the cost and brittleness of fine-tuning on knowledge that changes weekly. But the gap between a RAG demo and a dependable RAG system is enormous — and almost none of it is about the model.
Across the systems we've shipped, the same handful of decisions decide whether retrieval is accurate enough to depend on. Get them right and a mid-tier model feels brilliant. Get them wrong and the best model on the market still returns confident nonsense.
1. Ingestion is the real model
Most accuracy problems are ingestion problems wearing a costume. If your parser mangles tables, drops headings, or splits a sentence across two chunks, no amount of prompt engineering recovers the lost meaning. We treat parsing and chunking as first-class engineering: layout-aware extraction for PDFs, structure-preserving chunking that respects headings and lists, and metadata (source, section, date, permissions) attached to every chunk.
- Chunk on semantic boundaries, not a fixed token count — a heading plus its paragraph beats an arbitrary 512-token window.
- Keep a little overlap so context isn't severed mid-thought.
- Store rich metadata; you'll need it for filtering, citations, and permissions.
2. Hybrid retrieval beats pure vectors
Vector search is great at meaning and bad at exact matches — part numbers, error codes, names, acronyms. Keyword search is the opposite. In practice you want both: run vector and keyword retrieval, then combine and rerank the candidates. A dedicated reranker on the top results is often the single highest-ROI accuracy improvement we make.
The gap between a demo and a dependable system is almost always discipline, not models.
3. If you can't measure it, you can't ship it
The teams that succeed build an evaluation set from real user questions early — fifty to a few hundred questions with known good answers — and score every change against it. This turns 'the answers feel better' into a number you can defend. It also tells you the most important thing a RAG system can know: when to say 'I don't know' instead of guessing.
4. Respect permissions at retrieval time
The fastest way to lose trust is to surface a document a user shouldn't see. Filter by the requesting user's access before anything reaches the model — never after. Permission-aware retrieval isn't a feature you bolt on later; it's a property of the index.
The takeaway
Production RAG rewards the unglamorous work: clean ingestion, hybrid retrieval with reranking, an honest eval set, and permissions enforced at the source. Do those four things and accuracy stops being a mystery — it becomes something you can measure, improve, and trust.
