LASSO catches more SQL mismatches—but its headline benchmark is not an accuracy score

A new SQL-validation system reports a large numerical lead over four prior tools, but the number needs careful labeling. On the paper’s unified benchmark, LASSO-CC refuted 20,957 of 36,042 query pairs (58.15%). Plain LASSO refuted 16,074 (44.60%), while ParSEval—the strongest listed non-LASSO total—refuted 12,358 (34.29%). These are refutation counts: cases where generated test data made two queries execute successfully and return different results. They are not text-to-SQL execution-accuracy scores, and they do not prove equivalence for pairs the tools fail to refute. Source

Eight suites, one aggregate

The 36,042-pair set combines eight sources after sanitization: BIRD (10,153 pairs), Cubes (3,106), LeetCode (21,960), Literature (51), Calcite (221), XDataBM (343), TPC-H mutations (198) and LargeJoin (10). That breadth is useful, but it also makes the headline total sensitive to the largest component: LeetCode contributes about 61% of all pairs. LASSO-CC’s biggest gain over plain LASSO also comes there—11,279 refutations versus 6,984—because cross-checking generates datasets from both the reference and candidate queries. Source

The comparison is not fully symmetric. XData was not run on BIRD, Cubes, LeetCode or Literature because its implementation supports PostgreSQL while those suites use SQLite or MySQL. Gemini 3.8 Flash was omitted from most of the unified benchmark because of cost. Every pairwise tool received a 30-second timeout; XData received one minute to generate reusable datasets, while LASSO kept the 30-second limit. The paper also excludes runtime-error divergences and counts non-equivalence only when both queries execute successfully. Source

The generalization evidence is promising, but small

The authors say TPC-H and LargeJoin were unseen during development. LASSO refuted all 10 LargeJoin pairs and generated their datasets in about one second; Gemini also refuted all 10 but took 59.5 seconds and an estimated $0.053. On an unseen BEAVER enterprise-warehouse probe, however, the evaluation covers only 10 queries from one dw schema, not the full benchmark. LASSO produced non-empty datasets for 9 of 10 in 0.16 seconds; Gemini covered 10 of 10 in 43.4 seconds; ParSEval covered 5 of 10 and timed out on the rest. The paper explicitly says broader BEAVER conversion and testing remain ongoing. Source

LASSO is still a notable result: the authors publish the source, prompts, tests, experiment scripts and results, and the system’s many small heuristic “attack strategies” appear to trade solver generality for speed. But the clean takeaway is narrower than “better SQL accuracy.” LASSO-CC found more counterexamples on this mixed, sanitized benchmark under these tool-specific support and timeout rules. Independent reruns—and full-schema enterprise tests—are the next evidence to watch. Artifact