SQLoom gets a 1.7B model to 73.4% on BIRD dev—but the small model still needs a large teacher bill
The paper’s set-contrastive training beats its listed baselines, but 38,000 GPT-4 calls move agent cost from inference into dataset construction.
SQLoom reports 73.4% execution accuracy on BIRD development with Qwen3-1.7B, averaged over three seeds with a standard deviation of 0.5 points. That is a notable small-model result: in the paper’s comparison table, the next-best BIRD dev result is 68.9% from Arctic-Text2SQL-R1-7B. But this is a development-split paper result, not a BIRD test-leaderboard score, and the training pipeline still relies heavily on a proprietary teacher. Source
What changed in training
Instead of teaching one reference SQL string token by token, SQLoom contrasts sets of acceptable and unacceptable choices at three levels: syntax, question-to-SQL semantics and dependencies between clauses. A parser derives legal syntax; agents propose semantic confusions and counterfactual questions; execution on generated counterexample databases rejects pairs that only appear equivalent on the original data. Source
The ablation is the strongest evidence that the objective matters. Replacing the constraints with ordinary supervised fine-tuning on the same data cuts BIRD dev execution accuracy by 10.3 points. The final 73.4% result excludes BIRD’s external evidence; adding that evidence raises the paper’s result to 76.8%, a separate condition that should not be conflated with the evidence-free number. Source
SQLoom also reports 88.7±0.3% on Spider dev and 89.2±0.2% on Spider test. Its training-data audit flagged and manually corrected 378 Spider annotations, while all evaluation retained the official labels. The authors say training on the corrected Spider-Refined set improves official-set execution accuracy by 1.2 and 1.4 points in their two reported comparisons. Source
The cost did not disappear
The deployed model is compact and single-pass: the paper measures 0.42 seconds per query, or 23 times faster than its GPT-4-based CogSQL setup. Yet producing the training supervision required 38,000 GPT-4 calls, 10.8 GPU-hours and 21.3 GB of memory. SQLoom therefore shifts expense from every production query into a one-time training-data and verification pipeline; it does not remove the teacher dependency. Source
The practical result is promising but bounded. The paper evaluates Spider, BIRD and three Spider perturbation sets, then names nested queries and additional SQL dialects as future constraint targets. Until code, corrected data and independent reruns are available, 73.4% should be read as a split-specific research result—not as proof that a 1.7B model broadly replaces agent systems in production databases. Source
sources
comments · 0