DataTalk’s 102-question test puts a number on the value—and cost—of NL2SQL validation

A newly released open-source prototype offers a useful, unusually candid result for teams building natural-language database interfaces: validation can improve a small model’s execution accuracy, but it can also damage SQL that was already correct.

The result

DataTalk is a self-hosted Chinese-language NL2SQL application built around SQLite and an OpenAI-compatible model endpoint. Its evaluation report records 84 exact-result matches across 102 questions. The report displays that as 82.3%; the repository README rounds the same fraction to 82.4%. This article uses the underlying count where possible and the evaluation report’s displayed 82.3% when comparing rates.

The benchmark covers counting, salary statistics, grouped ranking, skill joins and multi-condition questions. Correctness is determined by comparing executed result sets with results from hand-written gold SQL rather than by comparing SQL strings.

The repository includes a same-model ablation. With semantic validation and automatic repair enabled, DataTalk records 84/102 exact-result matches, displays 82.3% in the evaluation report and says 16 questions were recovered after an initially wrong generation. With semantic validation disabled—but static SQL checks retained—the README reports 74.5% accuracy and three recovered questions. Comparing the displayed rates yields a 7.8 percentage-point difference.

Validation is not free

The more interesting disclosure is the failure cost. The project reports first-attempt accuracy of 66.7% with semantic checks, versus 71.6% without them. Its explanation is that semantic validation sometimes rejects a query that would have returned the correct result, after which the repair step can make it worse. In this run, better error feedback recovered more failures than the validator introduced, but the validator was not a monotonic safety layer.

DataTalk’s checks go beyond blocking writes. They verify referenced tables and columns, constrain known enumerated values, flag filters not requested by the user and require skill questions to use the join table. Rejected SQL is returned to the model with a specific reason for up to two repair attempts. Execution is also restricted to one SELECT or WITH statement through a read-only SQLite connection, with an eight-second timeout and a 200-row result cap.

What the number does not prove

This is author-reported evidence from a repository created on October 27, not an independent benchmark result. The test uses one domain, one 3B model and 102 questions over a job-listings database. The repository currently has no declared software license, despite being publicly accessible, and its Docker build is explicitly described as untested by the author.

The project’s own files also expose a reporting inconsistency worth preserving rather than smoothing over: the evaluation report displays 82.3%, while the README displays 82.4% for the same 84/102 count. The count is authoritative here; 84 divided by 102 is approximately 82.35%, so either one-decimal display can result from a rounding convention.

Those limits matter. Still, the ablation is more useful than a headline accuracy score alone because it isolates a concrete engineering choice. The practical lesson is not that every NL2SQL system should copy these rules. It is that validators should be evaluated for both rescues and false interventions. A team that only counts blocked bad queries can miss the correct answers its guardrail rewrites into failures.