text2ql’s 100% claim comes from synthetic harnesses—not random Spider and BIRD samples

The public artifact behind text2ql’s reported 100% execution accuracy does not run random samples from the official Spider and BIRD datasets. Its two benchmark scripts say they create 50-example synthetic datasets in Spider/BIRD file format, then build those databases and examples locally.

That conflicts with the accompanying paper, which calls the evaluation “50-query random samples from Spider and BIRD.” The distinction is material: these are reproducible package fixtures, but they are not inspectable benchmark samples and should not be compared with official Spider or BIRD results.

What the code establishes

The released run_llm_benchmark.py contains the corpus inline. It defines SPIDER_EXAMPLES and BIRD_EXAMPLES, plus compact database schemas and rows, then writes them into temporary Spider- and BIRD-shaped directories. The deterministic script imports the same two builders. Neither script selects rows from a downloaded benchmark, and neither exposes a source split, random seed or official example IDs.

The Spider-style side comprises 50 fixed questions across five named databases; the BIRD-style side similarly constructs 50 cases and its own local data. Calling the loaders with split="dev" describes the generated directory layout—it does not establish provenance from an official development split.

The released README is explicit: both score snapshots use “50 Spider-style + 50 BIRD-style synthetic examples.” On those fixtures it reports deterministic execution accuracy of 100% for Spider-style cases and 100% for BIRD-style cases, with zero errors. LLM completion with gpt-4o-mini is reported at 84% Spider-style execution accuracy and 90% BIRD-style execution accuracy.

The 90%/91% discrepancy

The paper’s Table 4 actually gives 90% BIRD execution accuracy for LLM completion, matching the README. Its 91% belongs to the BIRD function-calling row—not completion, as our earlier draft stated.

One inconsistency remains in the primary paper: a raw accuracy over exactly 50 cases can move only in two-point increments, so 91% cannot be the unrounded proportion of correct cases on one 50-item run. The paper does not describe averaging multiple runs or another denominator that would produce 91%. That number therefore remains unreconciled and should not be used as a benchmark result.

What can—and cannot—be concluded

The artifact clears one useful bar: readers can inspect the fixed questions, database contents and harness. It may be useful as a package regression suite. It does not support the paper’s description of random official benchmark samples, however, and it cannot establish performance on any named Spider or BIRD split.

nl2sql.ai is therefore not recording these percentages in its benchmark ledger. A comparable result would require running the released adapters against an official, named split and publishing the exact version, IDs or deterministic selection procedure, database files and scorer configuration. Until then, “100%” means all 100 synthetic package fixtures executed—not that text2ql scored 100% on Spider or BIRD.