DevRev reports 91.7% NL-to-SQL accuracy—but its benchmark is not public yet

DevRev researchers report 91.7% answer correctness, or 825 correct answers out of 900 queries, for a cost-aware NL-to-SQL agent evaluated against the company’s production Snowflake schema. The same paper puts the next-best baseline, FlexSQL, at 37.1% on that set—a 54.6-percentage-point margin. Those are striking numbers, but the evaluation has two boundaries readers should keep attached to them: the benchmark is not yet public, and the paper’s separate Spider 2.0 result is measured on a smaller subset than the public leaderboard.

What the 91.7% measures

The new DevRev NL2SQL benchmark contains 900 execution-verified questions spanning nested types and link-graph structure. The paper says the underlying schema has 13,898 fields, including 2,036 ARRAY[STRUCT] columns, with nesting up to four levels. Its answer-correctness metric is stricter than merely asking whether SQL executes: a query must pass deterministic checks, execute successfully and survive semantic validation by an LLM judge.

All compared systems use GPT-5.2-2025-12-11 as their primary model. On the private 900-query set, the proposed system scores 91.7%, versus 37.1% for FlexSQL, 29.1% for ReFoRCE and 27.2% for APEX-SQL. The authors also report $0.57 per correct query, compared with $15.35, $0.93 and $4.60 respectively, under their stated model-pricing and runtime assumptions.

The ablations make metadata retrieval—not model choice—the paper’s central finding. Replacing full metadata retrieval with type signatures alone lowers answer correctness by 15.0 points overall and 64.4 points on high-D7 questions. Replacing the structured error taxonomy with generic retry costs 13.2 points and raises mean repair rounds from 1.4 to 3.2.

The comparability limit

The paper says the 900-query benchmark “will be released publicly upon publication.” Until that happens, outside teams cannot independently reproduce its 91.7% result, inspect question construction or rerun the baselines on identical infrastructure. The authors also acknowledge that the set was LLM-generated and execution-verified rather than fully human-curated.

Its Spider 2.0-Snow result needs a separate label. DevRev reports 85.8% execution accuracy, or 103 of 120, on a public gold-answer subset. The public Spider 2.0-Snow leaderboard systems shown alongside it are evaluated on 547 questions. The paper explicitly says its 120-query score is not directly comparable with those full-set submissions.

That does not invalidate the architecture or the private-set result. It changes what has been demonstrated: a large gain under DevRev’s own nested-schema evaluation, plus a promising but partial Spider check. The benchmark will become substantially more useful when the 900 questions, scoring implementation and baseline runs are released together.