SQLMorph finds what BIRD’s binary score hides—and exposes its own mutation risk

A new ICDE 2026 paper argues that text-to-SQL evaluation should stop treating every wrong answer as equally wrong. SQLMorph mutates BIRD queries to stress joins and abbreviations, then scores partial outputs with execution precision, recall and F1. Its most useful result is diagnostic: systems that look similar under binary execution accuracy fail in measurably different ways. Its most important caveat is that the mutation process can introduce errors of its own. (SQLMorph paper)

Harder joins produce a sharp—but small-sample—drop

The authors evaluated CHESS, DIN-SQL and MAC-SQL on the BIRD development split only: 1,534 questions over 11 databases, not BIRD’s hidden test set. Join Query Expansion generated 6,873 executable variants, discarded 6,815 as structurally redundant, and retained a final 58-query expansion set. On those 58 queries, CHESS scored 44.83% execution accuracy, DIN-SQL 32.76%, and MAC-SQL 39.66%. Against their corresponding seed queries, CHESS fell from 63.33% to 44.83%, while DIN-SQL fell from 60.00% to 32.76%—a 27.24-point loss. MAC-SQL was nearly flat at 40.00% versus 39.66%. (paper, Table IV)

That difference matters more than a single leaderboard ranking. DIN-SQL’s recovery rate after receiving fixed context declined from 54.90% with one join to 24.05% with four joins among failed expanded queries. The result points to an interaction between retrieval context and generation difficulty rather than a uniform “more joins is harder” effect. (paper, Table VI)

The generator also needs evaluation

SQLMorph’s SQL expansion is graph-driven, but it uses an LLM to regenerate the natural-language question. A manual review of all 58 retained pairs found 43 fully aligned (74.1%), eight compatible but underspecified or noisy (13.8%), and seven misaligned or intent-losing (12.1%). That is too large to treat the stress set as automatic ground truth without review. The paper is appropriately transparent about the check, but the reported system scores mix genuine robustness failures with possible benchmark-generation failures. (paper, Join Query Expansion evaluation)

The abbreviation test raises a second reporting issue. The abstract says heavy abbreviation reduces accuracy by “up to 17%,” yet Table VII reports DIN-SQL changes as large as −23.9% when both natural-language wording and schema identifiers are made less natural. Readers should rely on the table’s defined conditions rather than the headline maximum. (paper, Table VII)

Partial credit becomes a debugging signal

On 410 BIRD-dev questions that all three systems failed under binary execution accuracy, SQLMorph’s semantic-column, partial-cell scoring separated them: DIN-SQL reached 66.26 precision, 22.05 recall and 19.71 F1; CHESS reached 51.44, 19.11 and 17.82; MAC-SQL reached 56.97, 15.10 and 13.82. All still had execution accuracy of zero. The paper macro-averages precision, recall and per-query F1 separately, so the displayed aggregate F1 is not the harmonic mean of the displayed aggregate precision and recall. These metrics do not excuse wrong SQL, but they reveal whether a system retrieves mostly correct material with extras or misses required results—information a binary leaderboard discards. (paper, Table XI)

SQLMorph is open source, making its real value testable: teams can mutate private workloads and inspect failure modes before deployment. The headline scores should remain labeled BIRD-dev-derived, 58-query stress-set results, not compared directly with hidden-test leaderboard numbers.