The model is often blamed too late
When a RAG or research agent gives a weak answer, the model is usually the first suspect. Sometimes that is right. But in many production traces, the model is answering from thin, stale, duplicated, or conflicting evidence. Better prompting will not fix a retriever that never found the relevant source, and a larger model will not reliably rescue context that points in the wrong direction.
Teams need to evaluate retrieval as a first-class part of the agent, not as an implementation detail hidden behind the final answer. The question is not only whether the answer was correct. The question is whether the agent had a fair path to the answer.
- Was the retrieved context relevant to the user question.
- Did the agent collect enough independent evidence to support synthesis.
- Did later search rounds improve the answer or only repeat earlier context.
Search rounds should show progress
Search is useful when each round changes the information state. A good second search narrows a vague question, checks a conflicting claim, finds a primary source, or fills a missing field. A bad second search repeats the same query, opens near-duplicate results, or collects more text without updating the working answer.
This is why retrieval evaluation should include time and sequence. Aggregate recall is not enough. You want to know when the decisive evidence appeared, how many rounds it took, and whether the agent recognized that it had enough information to answer.
- Query diversity across rounds.
- New evidence introduced per round.
- Synthesis checkpoints after material evidence changes.
- A stop decision once additional retrieval no longer improves confidence.
Evidence diversity beats volume
Long context can hide weak retrieval. Ten chunks from the same page are not the same as three independent sources that triangulate a claim. For enterprise data, diversity may mean different systems: CRM notes, support tickets, policy docs, billing records, and meeting transcripts. For web research, it may mean primary sources, recent updates, documentation, and trusted analysis.
An agent should be rewarded for collecting evidence that changes what can be known, not for filling the context window. This is especially important when agents perform multi-step work, because early retrieval mistakes can cascade into planning and execution.
- Track source family diversity, not only chunk count.
- Flag duplicate or near-duplicate evidence.
- Require primary or authoritative evidence for high-impact claims.
- Score whether the final answer uses the strongest retrieved evidence.
Synthesis checkpoints reduce drift
A synthesis checkpoint is a moment where the agent states what it knows, what remains uncertain, and what it will search next. It prevents search from becoming motion. It also creates an evaluation artifact: reviewers can see whether the agent understood the evidence before moving on.
In many failing traces, the agent searches repeatedly and only synthesizes at the end. By then, it may have mixed stale context with decisive context, or it may have lost the original question. A short checkpoint after important retrieval rounds keeps the run grounded.
- Summarize the current answer candidate.
- Name conflicts or missing facts.
- Decide whether another retrieval round is justified.
- Carry forward only evidence that supports the final answer.
What to compare before switching models
Before replacing the model, compare retrievers, chunking strategies, metadata filters, rerankers, and query generation. Keep the answer model fixed while you test retrieval changes. Then keep retrieval fixed while you test model changes. This prevents teams from attributing gains to the wrong layer.
A mature evaluation report should show retrieval metrics and answer metrics side by side. If retrieval quality is low, the model score is not the root cause. If retrieval quality is strong and the answer is still weak, then prompt, synthesis, or model capability is the right next investigation.
- Retriever recall and precision on representative tasks.
- Evidence diversity by task type.
- Search rounds until decisive evidence appears.
- Faithfulness between context and final answer.
- Cost and latency per successful answer.