MaP-SQL reports a 2.02-point BIRD-dev selector gain without fine-tuning

A new preprint reframes one expensive part of text-to-SQL systems: choosing the best query after a generator has already produced several candidates. MaP-SQL does not train a new selector. Instead, it retrieves structured memories derived from labeled training examples, ranks candidates listwise, and aggregates rankings across candidate permutations. On four fixed BIRD development-set candidate pools, the authors report an average 2.02 percentage-point execution-accuracy gain over their reproduction of R3-SQL, while using 2.92× fewer input tokens.

What was actually measured

This is a selector study, not a new end-to-end BIRD test-leaderboard result. The experiments hold candidate pools fixed and vary the method used to choose among them. The authors use two generators — Agentar-Scale-SQL-Generation-32B and Arctic-Text2SQL-R1-7B — with either eight or 32 candidates per question. Qwen3-Coder-30B-A3B-Instruct handles the listwise and pairwise comparisons, with Contextual-RM-32B used for optional tie-breaking.

The best BIRD-dev result in the main table is 73.08% execution accuracy for MaP-SQL selecting among 32 Agentar candidates. The reproduced, non-trained R3-SQL procedure scores 71.97% on that same pool. With 32 Arctic candidates, MaP-SQL reaches 72.62%, versus 69.56% for R3-SQL.

The efficiency difference is largest in that Arctic, 32-candidate setting. R3-SQL averages 216.59 selector calls and 510,881 input tokens per question; MaP-SQL uses 23.85 calls and 122,985 tokens. The paper describes those reductions as 9.07× in calls and 4.16× in tokens.

Why memory helps — and what “fine-tuning-free” means

MaP-SQL constructs memories from training-set question, schema and gold-SQL triples. Each memory records cues for schema grounding, SQL operations and output validation. At inference time, relevant memories are retrieved and added to the selector prompt. The system also groups candidates by execution result and permutes candidates within groups to reduce position bias without exhaustively comparing every pair.

An ablation on eight-candidate BIRD-dev pools supports both choices. Averaged across the two generators, removing permutation lowers execution accuracy by 0.32 points, while removing memory lowers it by 0.55 points. A separate order-sensitivity test raises selection consistency from 18.98% for basic listwise ranking to 33.17% after group-based permutation and tie-breaking.

The important qualification is that “fine-tuning-free” applies to the selector parameters. The approach still depends on labeled training examples to build its memory bank, pretrained models, multiple generated candidates and database execution signals. It therefore reduces training and inference overhead in a controlled selection stage; it does not establish a new production-system ceiling or a directly comparable BIRD test score.

The paper also reports transfer beyond BIRD: with 32 Arctic candidates, MaP-SQL records 87.59% execution accuracy on Spider-test and 44.71% on EHRSQL, beating the reproduced R3-SQL selector by 0.77 and 0.68 points respectively. The broader finding is narrower but useful: for fixed candidate pools, explicit retrieved criteria can replace some learned reranking behavior while costing materially less than dense pairwise selection.