BIRD-History turns old SQL into context—and exposes retrieval as the bottleneck

BIRD-History asks a practical question that standard text-to-SQL tests largely avoid: can a system recover business logic from SQL that analysts already wrote? The new benchmark contains 1,393 tasks across the 11 databases in BIRD’s development set, and its authors have released the dataset, annotations, retriever and evaluation scripts. It is a separate full-set evaluation—not a new score on BIRD’s hidden test leaderboard. Paper · Repository

What the benchmark changes

Each task supplies an underspecified question, the database schema and a same-database pool of historical BIRD-development queries. Ground truth identifies both the historical queries containing the needed knowledge and the SQL clauses that encode it. The paper says each task requires 5.3 historical fragments on average, with between one and 11 fragments reused. The released JSON includes the target SQL, relevant historical question IDs and clause-level retrieval annotations. Paper · Repository

The construction is partly synthetic. An LLM combines and modifies fragments from existing BIRD queries; validators check syntax, non-empty results and semantic consistency; three SQL-experienced authors then cross-validate every candidate. The paper reports 0.80 full agreement and Gwet’s AC1 of 0.82. That process makes retrieval measurable, but it is not evidence that the task distribution matches a production query log. Paper

The gain—and the remaining gap

On the full 1,393-task set, replacing DAIL-SQL’s masked-question retrieval with the proposed query-plus-component retriever raises execution accuracy from 40.78% to 51.33%, a 10.55-point gain. OpenSearch-SQL moves from 59.15% to 60.95%; N-rep from 54.49% to 57.79%; and CodeS-7B from 12.92% to 18.95%. All generation experiments in that comparison use the paper’s stated apparatus, including Qwen’s qwen-plus-2025-07-28 snapshot where an API model is required, temperature zero and retrieval restricted to the same database. Paper

The more useful result is the ceiling experiment. OpenSearch-SQL scores 54.27% with retrieval disabled, 60.95% with the proposed retriever and 67.34% when given gold history annotations. The implemented retriever reaches 83.07% recall but only 35.53% precision. In other words, historical SQL helps, yet choosing the right fragments remains at least as consequential as exposing the log in the first place. Paper

A fixed four-shot setting slightly beats the dynamic policy—61.31% versus 60.95% execution accuracy—while increasing precision from 35.53% to 38.75% and lowering recall from 83.07% to 79.33%. That is a warning against treating a larger context window as a retrieval strategy: extra examples can add competing but valid-looking SQL patterns. Paper

What to reproduce

The artifact clears the first reproducibility hurdle: code, data, requirements and run scripts are public. A serious reproduction should preserve the same-database retrieval boundary, report the full BIRD-History set separately from BIRD dev or test, and publish execution accuracy together with retrieval precision and recall. The next test is external validity: replace the curated historical pool with a noisy, evolving organizational query log and measure whether stale definitions, failed SQL and access-controlled fragments erase the reported gain. Repository · Paper