Tablo2SQL-TR shows why correct SQL can still return the wrong answer

A newly released Turkish document-table package makes a practical point for text-to-SQL teams: query correctness cannot rescue values that were parsed incorrectly before they reached the database. Tablo2SQL-TR combines an application, five test sets totaling 1,406 questions, and 5,012 question-level result records across a rule-based generator and six language models. Its source package is published with code, tests and reproduction instructions. Source

The locale is part of the data contract

The package normalizes Turkish number, date and unit conventions before loading extracted tables into SQLite. Examples include converting 1.234,56 to 1234.56, parenthesized values such as (1.200) to negative numbers, and Turkish month names to normalized dates. In the published experiment, the rule-based system reached 92.9% on TR-TabloSQL with Turkish-aware normalization. Under an English-locale assumption, even gold SQL returned the correct answer for only 28.4% of questions—and did so without raising an error. Source

That is the deployment lesson: teams should test the ingestion layer with locale-specific fixtures before using execution accuracy to judge the SQL agent. Otherwise a syntactically valid query can faithfully retrieve a silently corrupted value.

Stronger models expose the next bottleneck

The release also includes a real-document pilot covering 18 PDFs from seven institutions and 189 questions. The documents themselves are not redistributed, but the package lists their sources and supplies SHA-256 hashes for version checking. On that pilot, Claude Opus 5.5 scored 59.8% and GPT-6.1-sol scored 62.4%. When 32 questions whose values had not been read correctly into the database were excluded, the cloud-model range rose to 70.7%–73.9%. Source

The authors also injected 159 broken queries into an audit chain. They report that it detected 92.5% of the errors with no false alarms in that experiment. Those figures are package-reported, not an independent reproduction, and provider-default sampling means cloud-model reruns may vary. Source

For practitioners, the useful evaluation sequence is therefore broader than “did the model produce the expected SQL?” Verify the source document version, inspect extraction accuracy, test locale normalization, compare executed answers, and measure abstention or audit behavior separately. Tablo2SQL-TR is valuable less as a universal leaderboard than as a reproducible example of how preprocessing failures can dominate end-to-end NL2SQL quality.