BIRD’s corrected Mini-Dev resets the baseline—and makes MySQL the weak link

BIRD has replaced the working version of Mini-Dev after a human-in-the-loop review corrected reference SQL, clarified ambiguous output contracts and removed evidence that leaked unnecessary schema links, query plans or answers. The October 28 release contains 500 aligned question IDs in each of SQLite, MySQL and PostgreSQL—1,500 dialect-specific answers in all—and the project explicitly labels the July 2025 annotations and baseline tables as historical rather than current results.

A baseline reset, not a routine leaderboard update

That distinction matters. A score on the revised set should not be presented as movement against a run on the archived 2025 release: some annotations were corrected and selected instances were replaced. The benchmark’s own card says questions and evidence now match across dialects while the gold SQL remains dialect-specific.

On the new fixed-prompt evaluation, GPT-6.1 Sol leads the listed model rows at 87.27% overall execution accuracy, combining 88.00% on SQLite, 84.80% on MySQL and 89.00% on PostgreSQL. Claude Opus 5.5 is nearly tied at 87.13% overall, but with a different dialect profile: 90.00% SQLite, 84.40% MySQL and 87.00% PostgreSQL. Overall accuracy is calculated across all 1,500 dialect-specific answers, equivalent to the mean of the three 500-question scores.

The common weakness is MySQL. It is the lowest-scoring dialect for both leading systems, trailing GPT-6.1 Sol’s PostgreSQL result by 4.20 percentage points and Claude Opus 5.5’s SQLite result by 5.60 points. That pattern is not proof that MySQL is intrinsically harder, but it is a concrete target for error analysis because the questions and evidence are aligned by ID.

The remaining ceiling is not a human score

BIRD also reports a 96.80% empirical upper bound: 99.60% for SQLite, 93.80% for MySQL and 97.00% for PostgreSQL. This is an oracle-style union that counts a question correct when at least one evaluated model answers it correctly; it is not the performance of one deployable system. The 9.53-point gap between that overall union and the top single model therefore measures complementary successes as much as raw headroom.

The practical takeaway is methodological: rerun systems on the corrected artifacts and label the release. The revised Mini-Dev is useful precisely because it permits question-by-question dialect comparison, but its new scores must not be spliced into charts built from the archived annotations—or compared with BIRD’s separate main leaderboard as if the splits were interchangeable.