Provenance in RAG, and why it is usually wrong

Provenance is the record of which document said what. Every retrieval system claims to have it, and most have a version that works until two documents agree with each other. This guide walks through the designs in the order people usually try them, and where each one fails.

Attempt one: no provenance

You extract facts and store them as edges. Subject, predicate, object. Clean, small, fast.

It fails the first time anyone asks where a fact came from — which is immediately, because that is the only reason to build this instead of using a vector store. Worth mentioning only because it is a real intermediate state, and because a graph in this shape cannot be retrofitted: the information was never captured.

Attempt two: a document reference on the edge

Add a document identifier to the edge. Now every fact knows its source.

This fails the second time two documents assert the same thing. You have a choice, and both options are wrong:

• Overwrite the reference. The fact now claims to come from whichever document was processed last. Your citation is arbitrary, and re-ingesting in a different order changes it.
• Create a second edge. Now you have two edges between the same pair of nodes, which triples the apparent connectivity if a third document agrees, and makes any path scoring you do a function of how often something was said rather than how strongly it connects.

Attempt three: a reference set on the edge

Keep one edge and hang a set of document identifiers off it. Now the fact knows all of its sources, and the corroboration count is the size of the set. This is a real improvement and it is where a lot of systems stop.

What is still missing is per-document confidence. Extraction is not equally sure about every reading — one document states a relationship outright, another implies it in a subordinate clause. Collapsing those into one number per edge throws away the distinction between a fact resting on five confident statements and one resting on five hedges.

And there is a subtler problem: deletion. If a document is removed, you take its identifier out of the set. What if the set is now empty? You need a rule, and the rule has to be applied consistently, which means it wants to be a property of the data model rather than a line in a delete handler.

What to actually store

One row per (assertion, document) pair, each with its own confidence, plus the source system it came from and the vocabulary version that was active when it was written.

assertion_sources
  assertion_id      → the edge
  document_id       → what asserted it
  source_id         → which integration it arrived from
  ontology_version  → the vocabulary in force at the time
  confidence        → how sure extraction was, for THIS document

primary key (assertion_id, document_id)

With that table, four things become straightforward that were awkward before:

1. Corroboration is a count. How many documents support this fact is a query, not an estimate.
2. Retraction has a rule. A fact is retracted when its last supporting row is removed — not when any one document changes.
3. Evidence retrieval is a real query. Given a set of facts, find every document that supports them. This is what makes an evidence-restricted retrieval pass possible at all.
4. A vocabulary change is a filter, not a migration. Because each row records the version it was written under, recomputing conformance is a sweep over existing rows rather than a re-extraction.

Hydrate from every supporter, not one

A design decision that follows from the table and is easy to skip: when a fact contributes to an answer, pull passages from all of its supporting documents rather than one.

The cheap version picks the first or highest-confidence document per fact. It is faster and it loses the reason you built this. Three documents agreeing on a fact usually agree for three different reasons, and the useful one — the caveat, the exception, the number — is rarely in the first row returned.

The limitation nobody advertises

Corroboration counts are a floor, not a measurement, and you should know by how much before you put them in front of a user.

The reason is predicate variance. The edge identity includes the predicate, so a document saying "reduces" and one saying "lowers" produce two edges, each with one supporter, even though a human would call them the same claim. Lemmatising the predicate helps and does not solve it.

On our own corpus, 96.6% of edges have exactly one supporting document, and roughly 18.4% are predicate variants of another edge. So a displayed count of "1 source" frequently means "one phrasing of this claim was found", not "only one document says this". We show the counts because they are useful for ordering evidence, and we do not gate anything on them, because they are not accurate enough to gate on.

Provenance is not citation

One boundary worth drawing explicitly, because conflating these two produces a specific and nasty failure.

Provenance is internal: which documents assert which facts. Citation is external: the markers in an answer that a reader can follow. They are related but they are not the same, and a graph fact must never be offered as a citation.

The reason is that a fact is a derived object. "Onboarding reduces churn" is our reading of three documents, not a sentence any of them contains. If that appears as a citation, a reader who follows it finds a document that does not say what was cited — which is worse than no citation, because it looks checkable and is not.

The rule that follows: graph facts go into the synthesis prompt as context for how the sources relate, and every citation marker resolves to a document that was actually returned to the caller. Markers that do not resolve get dropped before the answer is handed over.

A checklist

1. One row per (assertion, document), with per-document confidence.
2. Retract a fact only when its last supporter is removed.
3. Record the vocabulary version on the row, so schema changes are filters.
4. Hydrate evidence from every supporting document.
5. Present corroboration as a floor, and gate nothing on it.
6. Never cite a fact. Cite documents.