QueryPilot hit 100%—then documented why the benchmark was too easy

A new open-source text-to-SQL project has published the kind of negative result that benchmark tables often hide. QueryPilot reports 100% execution accuracy for Claude Opus 5.5 and 99.2% for Claude Haiku 5.5 on its 123-question PostgreSQL test set, then immediately warns that the set is too easy to distinguish the two systems.

The perfect score needed an asterisk

The repository says Opus initially scored 92.7%. Its nine misses came from questions whose wording allowed more than one interpretation—for example, whether “how many reviews” meant rows or distinct review identifiers. The author reworded those questions without changing the reference SQL, after which Opus reached 123/123. That makes the published score useful as a regression check for this application, but not clean evidence that the model solved a previously fixed natural-language test unchanged.

QueryPilot at least exposes that intervention instead of burying it. Its README labels the result an upper bound and says the benchmark cannot yet demonstrate improvement because two differently priced models land almost together. The same table reports a full-run cost of about $1.33 for Opus and $0.04 for Haiku, with median latency of 1.9 and 1.7 seconds respectively.

Retries did not rescue quality

The project also tested its self-correction loop. Opus scored 123/123 without correction and 122/123 with it; the sole miss did not trigger a retry. On one previously failing question, correction recovered the answer in three of three live attempts, but the aggregate benchmark showed no measurable gain. That is a valuable boundary: an execution loop can repair SQL that throws an error or returns nothing, yet it cannot detect a plausible query that executes and answers the wrong interpretation.

The repository’s planned work acknowledges exactly that gap: harder naturally worded questions, saved traces, follow-up context and answer verification. Its scorer compares result sets rather than SQL strings, and the author reports checking it against 321 deliberately altered answers plus 6,020 cross-question swaps. Five swaps were accepted, attributed to identical outputs or the scorer’s extra-column rule.

What teams should copy

QueryPilot is not an independent model benchmark; it is a vendor-authored harness over one demo dataset, with one run per model and questions revised after observed failures. But its method is more reusable than its headline score. Keep the evaluation on the same pipeline users run, separate ambiguous-question repairs from model improvements, publish scorer false accepts, and record when a popular reliability feature produces no aggregate benefit.

A perfect score should end the test-design discussion, not the model-comparison discussion.