Text-to-SQL evaluation is finally moving beyond one accuracy number

A useful shift is showing up in new text-to-SQL repositories: builders are publishing not only an aggregate accuracy score, but also the failures that the score cannot see.

Three projects released on October 30 illustrate the change. Analytics Agent evaluates generated SQL on BIRD Mini-Dev, but it also separates syntactic or execution mistakes from business-definition mistakes. MinGuard treats privacy over-disclosure as a pre-execution decision problem. EvalForge publishes paired base-model and adapter predictions so readers can inspect both fixes and regressions.

Confidence is not governance

Analytics Agent reports that its full pipeline raised execution accuracy from 62.6% to 73.2% on 500 BIRD questions. Its more important result may be the smaller Olist experiment: raw SQL answered 14 of 30 governed metric questions correctly, definitions supplied in the prompt raised that to 27, and compiling governed metrics into SQL reached 29.

That experiment also exposes a limit of self-consistency. The repository says five of eight wrong-definition answers passed its confidence gate. Agreement among generated answers helped identify some SQL errors, but it did not reliably identify a consistently applied wrong business definition. The project also reports that confidence AUROC missed its pre-registered 0.70 target and that self-correction was not statistically significant run to run.

Privacy needs its own benchmark

MinGuard evaluates another surface that execution accuracy ignores: whether a database agent returns more personal information than the question requires. Its artifact includes five workloads, deterministic SQL fact extraction, recorded guard outputs and human labels for 350 requests across design and new databases. The labels distinguish added personal attributes, identifiers, unnecessary listings and extra people.

The repository provides scripts to regenerate its tables and figures without an API key or GPU because model outputs are recorded. That does not make the author-run result independently validated, but it makes the evidence inspectable and lowers the cost of reproduction.

Fine-tuning gains can hide churn

EvalForge reports that a Phi-3 QLoRA adapter raised SQLite execution accuracy from 53.0% to 57.5% on 200 held-out Spider development questions. The net gain is nine questions—but the paired record shows how it happened: the adapter fixed 30 base-model failures and introduced 21 regressions.

That distinction matters operationally. A four-and-a-half-point aggregate gain does not imply that previously working query patterns remain stable. EvalForge publishes all 200 predictions, hashes, an executed notebook and the training manifest, while explicitly describing the result as a development-set pilot rather than an official Spider test score.

A better minimum evidence package

Together, these releases suggest a practical evaluation floor for text-to-SQL tools:

• report execution accuracy on a named, versioned set;
• test governed business definitions separately from SQL generation;
• measure abstention and calibration rather than presenting confidence as certainty;
• test over-disclosure and read controls independently from correctness;
• publish paired predictions so improvements and regressions are visible;
• state when evidence is author-run and has not been independently reproduced.

None of the three projects yet has meaningful independent adoption. Their immediate contribution is therefore methodological, not market proof. But inspectable negative results are more valuable than a polished demo with one unexplained score—and they give future evaluators something concrete to rerun.