A “schema-free” text-to-SQL result still needs the schema for its public benchmark

A new Amazon-authored paper reports 95.7% execution validity on BEAVER’s 209-question evaluation, versus 60.8% for an embedding-retrieval baseline. That 34.9-point gap is substantial. It is also narrower evidence than the paper’s “schema-free” framing may initially suggest.

What the experiment actually tests

The proposed system mines historical SQL to build a graph of tables, business entities and join paths. On Amazon’s internal deployment, the authors processed about 500,000 historical queries across 240 frequently queried tables, producing 1,847 join edges. Two data engineers then blindly compared outputs on 35 questions—25 generated for coverage and 10 written by subject-matter experts—and preferred the graph system 70% of the time, versus 10% for embedding retrieval; the remaining 20% were ties.

That is the paper’s direct test of learning structure from query logs. But it is private, small and not independently reproducible.

The public BEAVER run uses a different construction. BEAVER supplies schema files rather than query logs, so the authors generated metadata from table names, columns, types and schema descriptions. They also inferred 1,113 implicit joins using column-name matching plus LLM validation, adding those to 127 declared foreign keys for 1,310 graph edges across 463 tables. The appendix explicitly says this adaptation has no usage-frequency information, may include synthetic relationships that never occur in real queries and lacks historical queries for few-shot examples.

Why 95.7% is not task accuracy

The headline public number is execution validity: whether generated SQL runs without error. It does not establish that 95.7% of answers are correct. In blind pairwise judging, the graph system was preferred in 45.5% of cases versus 34.0% for the embedding baseline, with 21 ties where both were adequate. A rubric judge preferred it 47.8% versus 30.6%, with 45 ties.

Those comparisons still support the retrieval contribution. The strongest dimensional gains were executability and schema validity, precisely where graph-guided table and join selection should help. An ablation also found that removing entity matching reduced the open-ended score by 5.4% and lost 43% of pairwise comparisons.

The defensible conclusion

This is promising evidence that entity-linked graph retrieval can assemble more executable SQL context than fixed top-12 embedding retrieval. It is not yet a reproducible demonstration that query logs can replace schema documentation: the only public evaluation starts from schema documentation, while the log-only evidence is an inaccessible 35-question internal set.

The next decisive experiment is straightforward: release a de-identified query-log corpus, freeze a temporal split so test questions cannot leak through historical SQL, and compare log-derived, schema-derived and combined graphs under the same generator. Until then, the result should be cited as a strong schema-derived retrieval result on BEAVER and a preliminary query-log result in production—not as one interchangeable benchmark claim.