Building a knowledge graph for retrieval

A knowledge graph built for retrieval is a different thing from a knowledge graph built for reasoning. You are not trying to model the world correctly; you are trying to store what your documents assert, with enough bookkeeping that you can find your way back to them. That difference simplifies some decisions and makes others harder.

Three primitives

You need exactly three things. Systems that need more are usually solving a modelling problem rather than a retrieval one.

• Entity — a thing mentioned in your content. Identity matters enormously and is the subject of most of this guide.
• Assertion — one edge per fact: a subject, a predicate, an object. One edge, not one per mention.
• Provenance — a separate record for each document that asserts a fact, saying that this document asserts it and how confident the extraction was.

The temptation is to collapse the third into the second by putting a document identifier on the edge. Resist it; that design breaks the second time two documents agree, and the breakage is subtle enough to survive review.

Entity identity is the whole problem

What makes two mentions the same entity? Decide that before you write a single edge, because everything else in the graph inherits the answer — and expect the rule to be tested hardest by the same word used for two different kinds of thing.

Resolution is the hardest thing here

Matching on the name catches most mentions and none of the interesting ones. It will not catch typos, near-synonyms, or two teams using completely different words for the same concept — and each of those wants a different and progressively more expensive kind of comparison, which is a project rather than a function.

Budget for it accordingly. This is the single hardest problem in building a retrieval graph, getting it wrong is invisible for months, and the cheap version works well enough in a demo to convince you it is finished.

Whatever you build, keep a human in the loop for the borderline cases. Merging automatically at a generous threshold quietly folds distinct entities together, and the symptom — answers that conflate two things — is very hard to trace back to a resolution decision made three months ago. An unmerged duplicate is repairable; a wrong merge usually is not.

The concurrency bug

This is the failure that everyone building this hits, and it does not announce itself.

Sequential extraction over a real corpus takes far too long, so you make it concurrent. Two workers now process two documents that both mention single sign-on. Both look for an existing entity. Both find none, because neither has committed yet. Both create one.

There is a correct answer here and it is not the one you reach for first. Budget for finding it, because nothing in your test suite will tell you that you have not: the failure only appears under real concurrency over a real corpus, and by then the duplicates are already written.

Assertions: one edge per fact

Every mention of a fact should strengthen one edge rather than create another. If three documents say onboarding reduces churn, you want one edge with three documents behind it, not three edges.

The reason is that traversal weights and corroboration counts both become meaningless otherwise. Three edges between the same pair of nodes makes that relationship look three times as connected as it is, which distorts any path scoring you do. And "how many documents support this" — the single most useful number a retrieval graph can give you — is only answerable if the edge is unique.

One caveat worth knowing before you trust your own counts: predicate variants defeat this. If one document says "reduces" and another says "lowers", you get two edges regardless, because the relation label is part of the identity. Read your own corroboration counts as a floor rather than a measurement — a count of one usually means one clear statement was found, not that only one document says it.

What NOT to store

Three things it is tempting to add and better to leave out.

• Inferred facts, mixed in with asserted ones. If you derive an edge from other edges, mark it as derived. Otherwise you eventually cite an inference to a document that never said it, which is the exact failure provenance exists to prevent.
• Hand-authored entities. An entity a person typed in has no provenance, and a fact with no document behind it cannot be cited. If you need editorial facts, keep them somewhere else and label them.
• Rich attribute payloads on the node. It is very tempting to hang metadata off entities. Retrieval does not read it, it drifts out of sync with the documents, and it turns your graph into a database with a maintenance problem.

The graph should be disposable

One question is worth asking of any design before you commit to it: if the graph store lost everything tonight, what would it cost to get it back?

If the answer involves re-running extraction, the slowest and most expensive step in the whole pipeline is sitting in the one place that can be lost, and a routine infrastructure failure becomes a model bill and a week of reprocessing. The same question applies to your vocabulary: if changing a term means re-extracting the corpus, nobody will ever change one.

A graph you can throw away and rebuild cheaply is one you can operate calmly. A graph you cannot is a second thing you have to keep backups of, and the backups have to be right.

A short checklist

1. Decide what makes two mentions the same thing, before you write a single edge.
2. Resolution has to survive concurrent writers, and it needs a review queue for the borderline merges.
3. One edge per fact. One provenance record per document behind it, each with its own confidence.
4. Mark derived edges as derived. Never store an entity a person typed.
5. Losing the graph should cost a rebuild, never a re-extraction.

None of this is exotic. All of it is the difference between a graph that answers questions after a year and one that has quietly filled with duplicates nobody can account for.