Governance & testing
How to evaluate a retrieval system honestly
Answer rate is the metric that flatters every system. Refusal controls are the one that catches the difference between grounding better and answering more.
11 min read
Most retrieval evaluations measure how often the system produces an answer. That metric rewards talkativeness, which means it will tell you your new architecture is better whenever it has merely become more willing to guess. This guide is about how to avoid that, and it is mostly about one control.
The metric that flatters everything
Assemble a hundred questions your corpus can answer. Run system A: it answers seventy. Run system B: it answers eighty-two. B wins.
Except you have not measured whether the twelve extra answers are correct, and you have not measured what either system does when the corpus cannot answer. A system that hallucinates confidently scores extremely well on this test. So does a system that has stopped refusing.
This is not a hypothetical failure mode. It is the default outcome of the most natural evaluation design, and it is why graph-RAG comparisons in particular should be read carefully — adding retrieval passes reliably increases the amount of context in the prompt, which reliably increases willingness to answer.
The fix: refusal controls
Include questions your corpus genuinely does not answer, and score a refusal as correct on those.
That single addition changes what the evaluation measures. A system that answers a refusal-control question has not demonstrated capability; it has demonstrated that it will produce plausible text without support. Now the metric distinguishes "grounds better" from "answers more", which is the distinction you actually care about.
A workable ratio is around a fifth of your set. We run twenty refusal controls in a set of a hundred and change. Fewer than about ten and a system can get lucky; many more and you are spending your evaluation budget on the negative case.
The four-cell table
Comparing two systems, the useful output is not two percentages. It is this:
| B answered | B refused | |
|---|---|---|
| A answered | Both answered — compare correctness | B lost an answer A had |
| A refused | B gained an answer A missed | Both refused — correct on controls |
The two off-diagonal cells are the whole result. "B gained an answer A missed" is the case for B. "B lost an answer A had" is the case against it. If both cells are near zero, the systems are equivalent on your corpus regardless of what the headline percentages say.
And on refusal controls, both cells being non-zero in the "answered" direction is a red flag rather than a win — it means whichever system answered a control question is hallucinating.
Retrieval metrics vs answer metrics
Measure both, separately, because they fail for different reasons and the fixes are different.
| Layer | Metric | What a regression means |
|---|---|---|
| Retrieval | Was a document containing the answer in the returned set? | Chunking, embedding, or ranking |
| Retrieval | At what rank? | Ranking or fusion weights |
| Generation | Did the answer state the correct fact? | The prompt, or context length |
| Generation | Did every marker resolve to a supplied source? | Retrieval returning too little |
| Generation | Did it refuse when it should have? | The prompt is not permitting refusal |
Separating these saves a lot of wasted effort. An answer that is wrong because the right document was never retrieved is not a prompt problem, and tuning the prompt will not fix it — but it will look like it might for a while.
Cheap diagnostics worth running continuously
A full evaluation is expensive and therefore periodic. These three are counters, and they run all the time.
- Invented-marker rate. Citation markers that do not resolve to a supplied source. A rise usually means retrieval is returning too little, not that the model got worse. Leading indicator for a retrieval regression.
- Refusal rate. Watched specifically for it approaching zero. A zero refusal rate is a bug — it means grounding has stopped working, not that your corpus became comprehensive.
- Context utilisation. What share of supplied passages ever get cited. If you hand over ten and two get used, you are paying for eight passages of context on every single request.
On graph-specific claims
If you are evaluating whether a knowledge graph helps, two things are worth measuring and they are not the same.
The first is participation: how often does graph-derived retrieval contribute a document to the final result set? This is easy to measure and it tells you whether the machinery is running. On our own corpus it is 201 of 231 question runs.
The second is contribution: does it answer questions the baseline could not? This needs the four-cell table above and a scored question set. It is much more work, and it is the only one that supports an accuracy claim.
Building the set
- Take real questions from your logs, or from support tickets, or from whoever answers them today. Invented questions are unrepresentative in a specific way: they are too well-formed.
- For each one, record the document that contains the answer. This is the tedious part and it is what makes retrieval measurable separately from generation.
- Add refusal controls — around a fifth of the set, plausible, adjacent to your domain.
- Include a few multi-document questions where the answer requires two sources. These are the ones a graph should help with, and if you have none, you have your answer about whether you need one.
- Version the set and never quietly edit it. A moving target produces improvements that are not real.
A hundred and twenty questions built this way is worth more than a thousand generated ones, and it is a couple of days of work rather than a project.