SchemaGate tops BIRD Mini-Dev in three dialects—but its rerun is the more useful result

SchemaGate has taken the top position on all three BIRD Mini-Dev dialect leaderboards with one Claude Opus 5.5 query per question: 81.0% execution accuracy on SQLite, 72.6% on PostgreSQL and 72.4% on MySQL. Mini-Dev contains 500 questions over 11 databases, and the project says each score was produced with BIRD’s own execution evaluator. The submission was accepted on October 31. (SchemaGate run report; BIRD leaderboard)

Those numbers improve on the leaders listed in the project’s October 19 snapshot: 75.6% for Jitto Build on SQLite, 65.8% for Ontology2SQL on PostgreSQL and 49.2% for TA + GPT-4 on MySQL. But the clean ranking is less informative than the artifact trail behind it.

The repeat run did not repeat the headline

SchemaGate reran the merged SQLite pipeline with the same named model and scored 80.2%, 0.8 percentage points below the submitted 81.0%. The repeat broke down to 89.86% on simple questions, 77.60% on moderate questions and 72.55% on challenging questions. One of the 500 questions returned no SQL because of an API error. The authors explicitly attribute the difference to nondeterministic model answers and publish both prediction sets. (SchemaGate run report)

That is not evidence that the leaderboard score is wrong. It is evidence that a single execution-accuracy number is a sample from a stochastic system. A 0.8-point swing is small enough to leave SchemaGate ahead of the prior SQLite leader, but large enough to reorder systems separated by less than a point.

Repairs did not create the 81.0%

The runner can show a database error to the model and permit up to three repairs. In the submitted SQLite run, however, 498 of 500 questions executed on the first query. The remaining two received one repair each and still finished incorrect, so execution accuracy before repair was also 81.0%. That log-level disclosure rules out a common ambiguity: whether the headline score came from repeated execution attempts rather than the first generated query. (SchemaGate run report)

The result remains self-reported and has not been independently rerun. It also uses BIRD’s standard evidence hints, sampled short text values and a frontier model, so it should not be read as the accuracy of schema pruning alone. The stronger precedent is methodological: publish predictions, evaluator output, per-attempt logs and a repeat run—even when the repeat is lower. BIRD’s next improvement would be to require several runs, or at least display variance beside every stochastic submission.