A Kazakh text-to-SQL model gains 40 points—then loses half the gain on a harder split

A new Kazakh text-to-SQL release makes an unusually useful distinction between familiar-schema performance and a harder test built from an unseen schema and independently written questions. The gap is large enough to change how practitioners should read the headline result.[^1]

The top line is 75.6%—on the easiest of four splits

The Apache-2.0 model starts from Qwen3-1.7B-Base and passes through three training stages: supervised fine-tuning on 3,744 question–SQL pairs, preference training on 1,568 pairs, and GRPO reinforcement learning on 1,650 questions. Its reward combines execution-result agreement with smaller bonuses for runnable, single-SELECT output.[^1]

On the project’s test_hero split, the base model’s three-shot execution match is 35.8%. Supervised fine-tuning raises that to 66.7%, preference training to 72.1%, and GRPO to 75.6%—a 39.8-point gain over the base.[^1]

But test_hero is held out by gold SQL from the training database. It tests new questions while preserving more of the environment the model learned.

The harder split cuts the result to 53.8%

On test_unseen_independent, which combines a database unseen during training with questions written independently from the training set, GRPO reaches 53.8%. That is still 20 points above the base model’s 33.8%, but it captures only about half of the 39.8-point gain visible on test_hero.[^1]

The intermediate stages also tell a useful story. On the independent test over a familiar schema, preference training reaches 73.8% and GRPO falls to 71.2%. On the unseen-schema split, GRPO is 67.8% versus 68.0% for preference training. The model card explicitly warns that differences of a few points may be noise because the test sets contain only 80–400 questions; it also discloses that four of 80 independent-test questions occur in training data.[^1]

What this release does—and does not prove

The associated dataset exposes 3,744 supervised-training rows and includes question text, gold SQL, query type, database identifier and result previews. Its examples span multiple small domains such as pharmacies, hospitals, libraries, shops and a Kazakhstan database.[^2]

The release is therefore reproducible enough to inspect, but its scope is narrow. The authors say schemas contain four tables and at most 200 rows, questions are model-written, SQLite is the only tested dialect, and real users with large enterprise schemas should expect lower accuracy.[^1]

The practical lesson is not that 53.8% is the “real” score and 75.6% is false. They answer different questions. Teams evaluating NL2SQL should report at least two axes separately: whether the schema was seen during training and whether test questions were authored independently. Without both, a large fine-tuning gain can mostly measure adaptation to a familiar synthetic environment rather than transfer to a new deployment.

[^1]: Hugging Face model card: Qwen3-1.7B Kazakh Text-to-SQL GRPO [^2]: Hugging Face dataset card: Kazakh text-to-SQL