SchemaGate’s 96.8% prompt reduction comes with the retrieval number teams cannot ignore

SchemaGate 1.14 adds a LlamaIndex retriever, but its more useful contribution is a reproducible measurement of a basic NL2SQL tradeoff: how much schema context can be removed before the system removes a table the answer needs.

The saving scales with warehouse size

The release’s token-savings script compares a prompt containing every column in a database with the tables selected by SchemaGate. It uses cl100k_base, calls no language model, and runs across BIRD dev plus Spider 2.0-lite schemas.

On BIRD’s relatively small databases—a median of seven tables—the selected schema reduced mean prompt tokens by 20.1%, from 579 to 462. The median saving per database was only 8.8%. On the 412 Spider 2.0-lite questions backed by BigQuery and Snowflake schemas, the mean fell from 46,560 tokens to 1,469, a 96.8% reduction. For the 33 databases with at least 50 tables, the median saving was 92.9%.

That is a practical result: schema selection matters much more in a production-sized warehouse than in the tiny databases that made early text-to-SQL benchmarks convenient.

The missing-table rate is the real boundary

Prompt reduction is not free. SchemaGate reports that its selected context contained every table referenced by the gold SQL for 98.4% of BIRD questions, but only 79.6% of the warehouse questions with public gold SQL. If retrieval drops a necessary table, no downstream model can recover it from the prompt.

The repository’s broader Spider 2.0 retrieval results reinforce that point. Across 247 questions, strict all-gold-table recall reached 80.6% at top_k=10 and 85.0% at top_k=20 with the default hashed embedder. Per-table recall at top_k=10 was higher, 91.3%, but partial recall is not enough when a join needs every relation.

The implementation includes a useful warehouse-specific repair: collapsing date-suffixed partition tables such as events_20201101 into a wildcard object such as events_*. With that behavior enabled, strict recall at top_k=10 rose from 72.1% to 80.6% in the published comparison.

What teams should copy

Version 1.14’s lesson is not to maximize token reduction. It is to tune schema selection against two measurements at once: context size and strict table recall. Large savings can make an agent cheaper and faster while quietly imposing a ceiling on answerability.

For production evaluation, teams should therefore record whether every required table reached the model, separate that failure from SQL-generation errors, and test physical warehouse patterns such as partition families. Schema retrieval is not merely prompt optimization; it is an upstream correctness gate.