NVIDIA’s synthetic SQL data lifts BIRD by 15 points—but its filter never runs the queries

NVIDIA says synthetic text-to-SQL training data raised Nemotron Super v3 from 26.77% to 41.80% execution accuracy on the 1,534-example BIRD dev split, averaged across five runs. That is a 15-point gain, and it puts the model above the 38.25% result NVIDIA reports for GPT-OSS-120B under the same comparison. But the more useful finding is in the pipeline’s rejection criteria: NVIDIA’s data filter checks SQL syntax and asks language models to judge quality; it does not execute each generated query to verify that it returns the intended result.

A large quality waterfall

The pipeline begins with 300,000 generated samples and retains 96,500, rejecting more than 68%. The final corpus is split roughly evenly across SQLite, MySQL and PostgreSQL, at about 32,000 records per dialect. Each example contains a natural-language request, a fully synthetic schema and a target query.

NVIDIA deliberately makes those schemas untidy. Each generated context contains 3–5 core tables, 1–2 distractor tables and 3–5 distractor columns per table. Generation prompts can also inject dates stored as text, currency symbols and JSON fields. The goal is to train schema linking and defensive SQL rather than reward a model for matching obvious table names.

The retained records pass a dialect-specific code validator and five LLM judges, each scoring dimensions on a 0–4 scale. That is substantially stronger than accepting raw model generations. It can remove malformed SQL, irrelevant joins and low-quality prompts.

Syntax is not semantics

The pipeline’s own “Next Steps” section names the gap plainly: its current waterfall validates syntax and uses judges, but does not verify that a query returns the right result. NVIDIA proposes adding a code sandbox that would execute generated SQL against a ground-truth database and compare outputs.

That distinction matters. A query can parse, use only declared columns and receive a favorable judge score while still joining on the wrong key, mishandling nulls or applying a business rule incorrectly. Conversely, NVIDIA notes that BIRD execution accuracy can credit semantically different queries when they happen to return the same result on limited data.

So the 41.80% BIRD dev EX result is evidence that the synthetic corpus helped this model on this split; it is not evidence that all 96,500 retained training pairs are semantically correct. The next defensible milestone is not simply a larger synthetic corpus. It is an execution-verified one, with failure counts reported separately for syntax, result mismatch and judge rejection.