When a retrieval-augmented system gives a bad answer, the instinct is to change the prompt or upgrade the model. In our experience, the majority of quality failures originate in retrieval. If the right passage never reaches the context window, no amount of prompt engineering will produce a grounded answer.
The fix is to make retrieval measurable and treat generation as the simpler, downstream concern.
Separate the two stages
A RAG system has two independent failure modes:
- Retrieval failure. The relevant passage is not in the returned context.
- Generation failure. The passage is present, but the model ignores or misreads it.
You cannot tell which you have without measuring them separately. Build a labelled set of questions, each paired with the passage that should be retrieved, then measure recall@k and mean reciprocal rank before evaluating any generated answer.
Start with chunking, not embeddings
Chunking is where most retrieval quality is won or lost. Fixed-size character splits cut sentences, separate a claim from its qualification, and produce fragments that embed poorly.
Better defaults:
- Split on semantic boundaries — headings, then paragraphs — before falling back to size limits.
- Overlap chunks slightly so context spanning a boundary survives.
- Attach metadata (document, section, date) so you can filter as well as search.
- Keep a parent reference so you can expand a retrieved chunk with its surrounding context at generation time.
Use hybrid retrieval
Pure vector search is bad at exact identifiers — part numbers, error codes, names. Pure keyword search is bad at paraphrase. Hybrid retrieval combines both and then reranks.
A practical pipeline: retrieve a broad candidate set with both lexical and vector search, merge the results, then apply a cross-encoder reranker to the top candidates. Reranking is often the single largest quality improvement available, and it is cheap relative to generation.
Treat retrieved content as untrusted
Retrieved documents are an injection surface. A malicious or corrupted document can contain instructions that hijack your model. Never let retrieved text change system behaviour, always delimit it clearly, and validate outputs against a strict schema before acting on them.
Measure groundedness, not vibes
For every generated answer, check whether its claims are supported by the retrieved context. Score it automatically where you can, sample and review where you cannot, and track the score over time.
Track four metrics together:
- Recall@k — did we retrieve the right thing?
- Groundedness — is the answer supported by the context?
- Answer relevance — does it address the question?
- Cost and latency per query.
A change that improves groundedness but triples cost is not obviously an improvement. Keeping these on one dashboard forces that trade-off to be explicit.
What to do when quality is still poor
If recall@k is high and groundedness is low, the problem is generation: simplify the prompt, constrain the output schema, or reduce the context to only what is relevant. If recall is low, no prompt change will help — go back to chunking, hybrid retrieval, and reranking.
Measure at the right layer first. It is the difference between a week of progress and a month of prompt roulette.