GraphRAG provenance works on 96.5% of 2Wiki cases—and 59% of MuSiQue

The number that matters is not answer accuracy

A new GraphRAG evaluation reports a result that ordinary question-answering scores miss: whether an answer can be traced through the executed query, the contributing graph statements and the exact source spans. In the GraphRAGProv paper, complete traces were available for 193 of 200 2WikiMultiHopQA cases (96.5%) but only 118 of 200 answerable MuSiQue cases (59.0%).

That 37.5-point gap is the main benchmark finding. The system uses DeepSeek-V4-Flash to construct a source-grounded RDF graph and generate SPARQL. The database—not a final-answer language model—returns the answer binding, while how-provenance polynomials identify which statement occurrences participated in the computation. Each occurrence links back to localized text spans.

A complete trace is a strict systems metric

The paper defines a complete trace as one in which every factor in the decoded derivation resolves to a graph-statement occurrence and matching source spans. Across the two 200-case splits, all 311 complete decoded traces passed the occurrence and source-span checks, and their answer bindings matched ordinary query execution. But 16 MuSiQue cases returned an answer without decoded provenance, showing that an answer can exist even when the promised audit path is incomplete.

The trace also distinguishes failure types that answer correctness can hide. One inspected case returns the right answer while attaching “Indian” to a film rather than its director: the offsets are real, but the interpretation is wrong. Another returns the correct year from two supported assertions without establishing the publisher relation required by the question. A missing-answer case contains the necessary facts but assigns the same person two different identifiers, breaking the join.

Those examples support a narrower claim than “provenance makes GraphRAG correct.” Provenance exposes the computation that actually ran; it can reveal that a correct answer came from an unsupported interpretation or an incorrectly expressed query.

The evaluation is diagnostic, not a deployment score

The authors explicitly used a supervised setting. Graph construction and SPARQL generation had access to the benchmark question, expected answer and reasoning evidence, while document retrieval was excluded by supplying the designated supporting passages. The system retained the first query that validated mechanically and executed, without retrying based on correctness.

That setup makes the 96.5% and 59.0% figures useful measurements of trace availability under controlled conditions, not estimates of end-to-end performance on unseen questions. The large benchmark gap is still informative: an audit mechanism should be evaluated as a pipeline with its own coverage metric, separate from answer accuracy. A system that produces correct answers without complete traces cannot claim complete auditability—and GraphRAGProv now gives evaluators a concrete way to count that shortfall.