BIRD marks some valid SQL answers wrong because one question can encode two calculations

One question, two executable answers

A text-to-SQL system can return runnable SQL and a plausible number while silently choosing among calculations the user never distinguished. The new SpecBridge study turns that problem into a controlled benchmark: each of 204 hand-reviewed request sets contains one open request and two otherwise matched requests that explicitly select interpretation A or B. All three run against the same real database, and execution—not an LLM judge—determines SQL correctness.

The distinction matters because the four evaluated models—Claude Haiku 4.5, Claude Sonnet 5.5, GPT-6 Sol and GPT-6 Luna—usually knew how to compute both readings when each was requested separately. Yet among sets where a model could compute both, its open-request answer returned both interpretations only 15% to 58% of the time. In 72% to 94% of missed cases, the same model produced the omitted interpretation correctly when asked for it alone. That makes the dominant error a reporting failure rather than a SQL-generation failure.

The benchmark score can reward convention, not correctness

The paper then audits BIRD development questions whose wording leaves a calculation choice open. It identifies 156 questions—about one-tenth of the development split—whose result changes under the alternative interpretation. After author review, 79 questions retained two valid interpretations. Across the four models, accepting either executable answer raises accuracy on that subset from 25% to 57%.

This is not a revised overall BIRD leaderboard score. It is a targeted result on 79 ambiguous development questions, pooled across four models, and it must not be compared with BIRD test-leaderboard execution accuracy. Its value is diagnostic: a single-reference benchmark can classify a defensible answer as wrong merely because it differs from the dataset author’s convention. The paper reports 17 BIRD questions on which all four models chose the alternative interpretation that the authors confirmed was valid.

The controlled set also exposes where systems are weakest. Claude Sonnet 5.5, the strongest reporter in this study, returned both readings for 68% of aggregation-level cases but only 43% of counting-unit cases and one of 22 tie-handling cases. No evaluated model returned both interpretations for more than one tie case. Hidden choices inside LIMIT and COUNT are particularly easy to miss because the natural-language request may never name them.

A better evaluation target

SpecBridge’s proposed remedy is “decide then answer”: first classify whether the request leaves a choice open, then answer each interpretation in a separate call. On held-out AMBROSIA questions and SpecBridge request sets, that procedure increased full-interpretation coverage by 28.9 and 63.1 percentage points, respectively, over a single production-style answer, without a measured precision loss on clear requests.

The broader lesson is narrower than “multiple references fix text-to-SQL.” Benchmarks need references that represent genuinely valid interpretations, plus a penalty for inventing alternatives when wording is clear. SpecBridge provides executable pairs for that trade-off. Until leaderboards adopt it, exact execution accuracy can still confuse agreement with the annotator for agreement with the user.