The 2026 production RAG stack

Retrieval is where most production AI actually lives, and where most of it quietly underperforms. The model is rarely the bottleneck; the chunk it was handed usually is.

Chunking is a retrieval decision, not a preprocessing one

Fixed-size chunks split arguments in half. Structure-aware chunking — by heading, by clause, by row — keeps the unit of meaning intact, which is what a query is actually looking for.

Hybrid beats pure vector, still

• Dense retrieval handles paraphrase and intent.
• Sparse retrieval handles the exact product code, error string or surname the user typed.
• Fusing both, then reranking the top candidates, consistently outperforms either on real query mixes.

Permissions belong in the index

Filtering results after retrieval leaks in two ways: through result counts, and through the model summarising what it should not have seen. Attach the access predicate to the query so restricted documents never enter the candidate set.

Citations are a correctness feature

Quoting the source is usually framed as a trust feature for the reader. It is also the cheapest way for you to notice the system is confidently retrieving the wrong document — a claim with a citation you can click is a claim you can falsify.

What to measure

1. Retrieval recall at the k you actually pass to the model — not at 100.
2. How often the answer is grounded in a retrieved chunk versus invented around it.
3. Latency at p95, because retrieval is where the tail lives.