FEBRUARY 2026
What I Got Wrong About RAG
Three retrieval mistakes I made building enterprise RAG systems — and the fixes that actually moved quality.
I have built and operated RAG systems across manufacturing configurations, validation assets, and enterprise knowledge bases. Here is what I got wrong, in the order I got it wrong.
1. I treated chunking as plumbing
Fixed-size chunks with a little overlap — how hard can it be. The answer: it breaks the moment your corpus is not prose. Enterprise knowledge is tables, configuration blocks, structured definitions. Cutting through those boundaries produces chunks that retrieve confidently and mean nothing.
The fix: structure-aware chunking that respects the source’s own boundaries. It is the single highest-leverage change I have made to any RAG system, and it is usually free.
2. I shipped pure vector search
Embeddings are wonderful at “means something like” and hopeless at “is exactly”. Our corpora were full of identifiers — error codes, part numbers, standard IDs — and vector search missed them while semantically waffling nearby.
The fix: hybrid retrieval (vector + keyword) with metadata filters. If your corpus contains IDs, this is not an optimization; it is a requirement.
3. I let the model decide what wins
When site-specific rules conflicted with global standards, retrieval returned both, and the model improvised precedence per answer. Auditors loved that. (They did not.)
The fix: encode precedence structurally — tiered retrieval where site-specific context shadows global context by design. Conflicts resolve deterministically; humans can read the rule; the model never has to guess.
The pattern behind all three
I kept trying to fix with generation what was broken in retrieval. When quality dropped, the instinct is to tweak prompts. The productive question is always: “is the right context even reaching the model?” Measure retrieval in isolation, with labeled examples. It is unglamorous, and it is where the quality lives.
What I do now
Golden set first, retrieval metrics per layer, hybrid search by default, precedence as structure, and a prompt that says “if the context does not contain the answer, say so.” Users forgive “I don’t know.” They never forgive confident wrongness.