Core concepts
Graph RAG vs vector RAG
A straight comparison: what each one retrieves, what each one can state, where the latency goes, and how to decide without running a bake-off.
10 min read
Vector RAG retrieves passages that resemble the question. Graph RAG does that too, and additionally retrieves passages connected to the question through facts extracted from your corpus. The second is a superset of the first, which makes the comparison less about quality and more about whether the extra work pays for itself on your content.
What each one actually retrieves
| Vector RAG | Graph RAG | |
|---|---|---|
| Retrieval passes | One | Typically three |
| Matches on | Semantic similarity to the question | Similarity, plus graph adjacency, plus provenance membership |
| Finds documents sharing no vocabulary with the query | Rarely | Routinely |
| Can enumerate the documents behind a claim | No | Yes |
| Needs a model call at ingest | No — embedding only | Yes — extraction per document |
| Needs a model call at query time | No | Yes — entity extraction from the question |
| Added query latency | None | Typically 400–700 ms |
The one question type that separates them
Almost every question either system handles equally. The separation happens on questions whose answer requires a relationship that no single passage states.
Take "do I need the security module before the deployment lab?" The lab document lists a prerequisite: familiarity with role-based access control. The security course teaches role-based access control. Neither document mentions the other course by name.
Vector RAG will likely retrieve both, because both are topically adjacent to the question. What it cannot do is state that one is a prerequisite for the other, because that sentence does not exist in the corpus. So the model either hedges, or it asserts the connection and cites two documents that do not actually say it.
Graph RAG has an edge — deployment lab, prerequisite, role-based access control — that a document did assert. The connection is retrieved rather than inferred, and the answer can state it while citing the documents that each hold half.
Where the latency goes
Worth being concrete, because "graph RAG is slower" is not actionable.
| Step | Typical | Notes |
|---|---|---|
| Direct hybrid retrieval | ~140 ms | Identical to vector RAG. Runs regardless. |
| Entity extraction from the query | ~100 ms | A small model call. Cacheable for repeated queries. |
| Two-hop traversal | ~190 ms | Corpus dependent. Grows with graph density, not size. |
| Expanded retrieval pass | ~130 ms | A second embedding and search. |
| Evidence retrieval pass | ~200 ms | Restricted to provenance documents. |
| Rank fusion | ~3 ms | Negligible. |
| Answer synthesis | ~900 ms | The same in both systems, and the largest single cost. |
The useful observation is that synthesis dominates. If you are generating an answer at all, the graph overhead is roughly a 50% increase on a number that was already close to a second. If you are only returning passages and not generating prose, the graph overhead is proportionally much larger and the calculus changes.
What graph RAG does not fix
Three things people expect it to solve and it does not.
- Content that does not contain the answer. If nobody wrote it down, no retrieval architecture will find it. This is by far the most common cause of bad answers, and it is a content problem.
- Bad chunking. If your chunk boundaries cut claims in half, both systems retrieve half-claims. The graph will faithfully extract facts from mangled text.
- Ranking preferences. If your users want the most recent document and your retrieval prefers the most similar one, that is a ranking policy question. A graph does not know what you consider authoritative.
How to decide without running a bake-off
A proper evaluation is expensive — you need a question set with known answers and a scoring rubric. Before committing to that, three cheap signals get you most of the way.
- Count your content sources. One source with one vocabulary means the graph has nothing to bridge. Two or more, using different words for the same concepts, is where it earns its keep.
- Sample twenty real user questions and mark each one: answered by a single passage, or requiring two. If almost all are single-passage, you have a lookup problem and vector RAG is sufficient.
- Ask whether anyone will need to justify an answer. If a wrong answer is embarrassing rather than expensive, the evidence trail is a nice-to-have. If someone will be asked "where did this come from", provenance is the feature you are actually buying.
An honest note on the accuracy claims
Graph RAG is currently fashionable, and a lot of published comparisons are not measuring what they claim. The specific trap is counting answer rate: a system that answers more questions looks better on a naive metric, even if some of the extra answers are wrong.
The correction is to include questions your corpus genuinely cannot answer, and score refusal as correct on those. Without that control, "answers more" is indistinguishable from "grounds better", and the more talkative system wins regardless of whether it is right.
The short version
- Choose vector RAG for lookup questions, a single source, tight latency budgets, and small corpora.
- Choose graph RAG for questions that span documents, content spread across several systems, and any situation where an answer has to be defensible.
- Whichever you choose, insist that the vector baseline always runs. A graph that can degrade your floor is a liability.
- And measure with refusal controls, or you will measure the wrong thing.