Research
What we measured, and what we haven't
Our evaluation harness, the refusal controls, the graph participation rate, and the plain statement that graph retrieval has not yet been shown to improve answer accuracy over plain RAG on our question sets.
8 min read
Every product in this category publishes a benchmark table. This page is the reason we do not, and what we publish instead. It is deliberately the least flattering page on the site, and it is the one we would most want to read if we were evaluating us.
The headline result
So the machinery runs, on the large majority of searches, and we cannot demonstrate that it makes the answers more accurate. Both halves of that are true and both are worth saying.
What "participates in fusion" means
A search runs three retrieval passes: a direct hybrid search on the query, an expanded search using entities reached by traversing the graph, and an evidence search restricted to the documents that provide provenance for the facts crossed. The three result sets are merged on rank.
Participation means at least one document in the final fused set came from the expanded or evidence pass rather than only from the direct pass. It is a real measurement of whether the graph is doing anything, and it is not a measurement of whether that something is useful.
Why answer rate is not the metric
The natural way to evaluate this is to ask a hundred questions and count how many get answered. It is also the way to fool yourself, because adding retrieval passes reliably increases the amount of context in the prompt, and more context reliably increases a model's willingness to answer.
A system that has become more willing to guess scores better on answer rate while being less trustworthy. So answer rate cannot distinguish "grounds better" from "answers more", which is precisely the distinction that matters when the claim is about a knowledge graph improving quality.
Refusal controls
The correction is to include questions the corpus genuinely cannot answer, and to score a refusal as the correct output on those. We run twenty of them in a set of a hundred and change.
They are written to be plausible rather than absurd: adjacent to the domain, phrased like a real question, about something the corpus nearly covers. A control question that no retrieval system would find anything for is not a test — the interesting case is when retrieval surfaces something close and the system has to notice the gap.
This is the single most important part of the harness, and it is the part that produced the uncomfortable result above. Without it, the graph configuration would have looked better.
The four-cell comparison
Comparing two configurations, we do not compare two percentages. We compare the off-diagonal cells:
| Graph answered | Graph refused | |
|---|---|---|
| RAG answered | Both answered — compare correctness | 1 question — graph lost an answer |
| RAG refused | 0 questions — no answer gained | Both refused — correct on controls |
The bottom-left cell is the case for the graph, and it is empty. The top-right is the case against, and it has one question in it. On this evidence the two configurations are equivalent in accuracy, with a marginal edge to RAG-only.
What we changed because of this
Two things, both worth mentioning because they show the harness earning its keep rather than sitting in a drawer.
First, corroboration no longer multiplies the fused score. It was intuitive that facts supported by more documents should rank higher. Across 2,160 evaluated configurations, that was the single worst one — popular facts crowded out correct ones, because corroboration measures how often something is said rather than whether it answers the question. Corroboration now orders the evidence pass and contributes to path scoring during traversal, and touches the final ranking not at all.
Second, we stopped claiming an accuracy improvement. An earlier internal design document carried a figure of "80% better accuracy on complex questions". It was never substantiated by any measurement, and it does not appear anywhere in the product or on this site. The shipped upgrade prompt in the console says only "far better accuracy on multi-hop questions", which is a claim about capability rather than a number, and even that is on our list to revisit.
Known limitations of our own numbers
Three, in descending order of how much they should change your reading.
- Corroboration counts are a floor, not a measurement. On our corpus 96.6% of edges have exactly one supporting document, and about 18.4% are predicate variants of another edge — the same claim phrased differently and therefore not folded. So "1 source" often means "one phrasing was found", not "only one document says this".
- Relationship labels are display-only. Traversal filters on provenance and corroboration, never on the predicate string. Diagrams that show labelled edges driving retrieval would be misleading, so we say this on the retrieval-engine page too.
- Our question sets are ours. They are built from a corpus we have, which means they under-represent the multi-source, multi-vocabulary case that the graph is theoretically best at. A set drawn from a genuinely fragmented corpus might produce a different result. That is a reason to be uncertain, not a reason to assume the number would improve.
So what is the graph for?
On this evidence, three things — none of which is a percentage point of accuracy.
- Stating a relationship no single passage contains. The graph holds an edge that two documents each half-support, and an answer can state it while citing both. A vector store can return both passages and cannot state what they add up to.
- The evidence trail. Every fact enumerates the documents that assert it, each with its own confidence. That is what makes an answer defensible when someone asks where it came from.
- A governable vocabulary. Versioned, with impact previews, reversible applies and drift detection. This is genuinely unusual and it is the pillar we would lead with.
If what you need is a measurable accuracy improvement over plain retrieval, we cannot yet show you one, and the free RAG tier is the honest recommendation. If what you need is to defend an answer or to govern a vocabulary, those we can demonstrate.
Why publish this
Partly because it is true and someone would eventually run the comparison themselves. Mostly because a category where everyone publishes a favourable chart is a category where charts have stopped carrying information, and being the one that publishes the unfavourable result is the only credible move left.
If you are evaluating us and this page makes you less likely to buy, it has worked correctly. We would rather that happened now than in month four.