Compiled claims match a heavyweight retrieval stack at roughly one-twentieth the query tokens

A new author-run benchmark makes a useful distinction that headline accuracy can hide: retrieval payloads can be dramatically cheaper even when they are not more accurate than the strongest alternative.

The study evaluates ingest-time semantic compilation (ISC) on a held-out set of 499 questions over 500 MediaSum broadcast-interview transcripts, with no document overlap with the extraction-development sample. Instead of retrieving raw chunks, ISC extracts atomic claims at ingestion and stores each claim with its source quote, speaker and location. The validation gate admitted 69,746 claims; in a 20-document replay, it rejected 29 of 2,724 candidates (1.1%), including 28 whose quoted evidence could not be located in the source.

The decisive comparison is a tie

At a nominal 2,048-token read budget, compiled claims answered 85.2% (425/499) correctly while sending about 2,200 tokens down the reader path. A contextualized-chunk pipeline—hybrid dense and full-text retrieval, reciprocal-rank fusion and a cross-encoder reranker—scored 88.0% (439/499). The paired McNemar test found no significant difference (p=0.202).

That tie matters because the stronger chunk stack consumed about 47,700 query-path tokens, roughly 21 times the compiled-claim path. At the 16,384-token setting, the stack’s 87.4% versus claims’ 83.6% was also not statistically significant (p=0.076). The defensible result is therefore not that compilation beats the best retrieval pipeline on accuracy; it is that it reaches statistically comparable accuracy with far less query-time context.

Against ordinary chunk policies, the result is clearer. Claims scored 85.2% with about 2,200 reader tokens, while the best fixed-width, turn-aware or semantic chunk configuration reported anywhere in the sweep reached 72.5% with about 16,300 tokens. Claims at only 256 tokens still scored 69.5%, beating every ordinary chunk policy at a 2,048-token budget. All 24 paired facts-versus-chunks comparisons survived Holm correction, with p≤4.9×10⁻⁶.

The cost moves to ingestion

Compilation is not free. Claim extraction used Kimi K2.6 and averaged 48,200 prompt plus 4,500 completion tokens per document, estimated at $0.064 per document. For the 500-document corpus, the paper estimates about $32 and 26.3 million tokens up front. Its cost model places break-even against the contextualized stack at roughly 580 reads, or about 600 after a replay adjustment.

The limitations should travel with the number. This is an author-reported study, not an independent reproduction. Automated answer grading also used Kimi K2.6, the same model family used for claim extraction; human calibration and cross-model regrading remain future work. The maintenance experiment—33.7× cheaper incremental updates than full reconstruction—used synthetic embeddings and is explicitly described as an idealized pilot.

For production teams, the benchmark suggests a measurable decision rule: compare total ingestion cost plus about 2,200 tokens per read against a query-time retrieval stack near 47,700 tokens per read. But do not turn the statistical tie into an accuracy victory. The evidence supports a token-efficiency claim first.