CypherGuard reaches 88.50% on CypherBench—but non-empty wrong answers remain its blind spot

CypherGuard reports 88.50% execution accuracy on the released 2,348-question CypherBench test set, up from 63.42% for the same GPT-5.4-mini backbone with the benchmark’s zero-shot prompt. That is a 25.08-point gain, measured across seven test graphs whose vocabularies are disjoint from the four training graphs used to build the system’s few-shot corpus and rule library. The paper says accuracy improved on every test graph, with per-graph gains ranging from 20.70 points on movie to 31.49 points on nba.

The result matters because CypherGuard is not another fine-tuned query model. It is an inference-time layer that drafts a query, applies deterministic schema and storage checks, then invokes an execution-driven repair loop when the query errors or returns no rows. The authors mined 14 preventive rules from GPT-5.4-mini failures on CypherBench’s training graphs and retrieved examples from 8,534 training pairs. Each test question uses one draft call and can trigger at most three repair calls, according to the paper.

The strongest evidence is not just the headline score. A drop-one ablation attributes the largest contribution to the execution loop: removing it lowers accuracy by 10.78 points. Removing few-shot examples costs 6.60 points, removing the rule library 4.39 points, disabling literal grounding 2.34 points, and removing shape examples 1.24 points. The same rule library, mined from GPT-5.4-mini failures, was reused with Claude Sonnet 4.6; that backbone rose from 71.76% to 94.68% on the same CypherBench test set.

A second dataset makes the transfer story more cautious. On an audited 1,003-question, 11-database subset of Neo4j Text2Cypher, the authors rebuilt the examples and rules from that dataset’s own training split. GPT-5.4-mini improved from 47.36% to 56.33%, while Claude Sonnet 4.6 rose from 56.63% to 62.51%. Those gains are real, but much smaller than on CypherBench and do not establish that one fixed library transfers unchanged across datasets.

The paper’s most useful finding is also its limit. GPT-5.4-mini’s zero-shot queries were already 93.52% executable, despite only 63.42% being correct. CypherGuard raised executability to 99.83% and provenance-subgraph Jaccard similarity from 0.775 to 0.933, yet still missed 270 questions. Of those failures, 52.2% came from the fictional_character and politics graphs. The authors identify the central blind spot plainly: a query that runs and returns plausible but wrong rows may trigger none of the guards.

That makes 88.50% a strong correction result, not a solved-reliability result. The benchmark shows that database execution is valuable feedback; it also shows that “returned rows” is too weak a certificate of correctness.