OCTOPUS cuts semantic-query judgments 10×—and shows where reuse can hurt quality

A new semantic-query prototype reports a large reduction in model work by saving model judgments as database data rather than discarding them after each query. In the eight-query movie scenario from SemBench, OCTOPUS sends 268 rows or row-pairs to Gemini 2.5 Flash, versus 2,768 for iPDB, 17,240 for ThalamusDB, 17,038 for Palimpzest and 28,910 for LOTUS. The paper describes those totals as reductions of 10×, 64×, 64× and 108×, respectively (paper, Section 5.2).

Those are not latency or dollar-cost measurements. The authors deliberately count the number of rows or pairs judged by the model, independent of pricing and batching. All five systems use Gemini 2.5 Flash over the same SemBench movie setup: 1,865 reviews and 116 movie rows including duplicates. Where a system packs multiple pairs into one prompt, the study counts the pairs carried by that prompt (paper, Section 5.2).

The clean comparison is 10×, not 108×

The most informative baseline is iPDB. Both systems delegate relational work to a database and avoid duplicate work within a query. OCTOPUS’s 268-to-2,768 advantage therefore isolates the benefit of carrying judgments across queries: about 10.3× fewer judged rows by direct division. The larger gaps also include plan and join differences, so presenting 108× as a pure caching gain would overstate the result (paper, Section 5.2).

OCTOPUS persists each verdict as a tag with provenance, including model, timestamp and confidence. Later queries can reuse an exact predicate, its complement or a taxonomy relationship. Three of the eight benchmark queries require no model call because prior queries have already filled the needed tags (paper, Sections 2 and 5.2).

Reuse also preserves mistakes

The benchmark exposes the corresponding quality risk. Query Q6 inherits tags written by Q5 and scores 0.58. Q7 judges the remaining reviews instead and reaches 0.74. The authors propose confidence thresholds and stricter predicate matching as controls, but explicitly leave trustworthy guarantees and tag maintenance as open problems (paper, Sections 5.2 and 6).

The result is best read as a workload study, not a general leaderboard win. SemBench executes each query once in a fixed order, and the paper evaluates one movie scenario with one model. Still, it quantifies an important systems tradeoff: persistence can turn repeated semantic work into ordinary SQL, while also turning one model error into reusable database state (paper).