The context library beat the agent harness in a 5,176-query production SQL study

Enterprise text-to-SQL teams often tune the agent around the model: prompts, retrieval tools, search strategy and repair loops. A new production study points to a different first target—the material being retrieved.

The paper evaluates 5,176 production queries drawn from a major online retailer’s warehouse and a corpus of roughly 100,000 historical query profiles. It turns those histories into per-table “SQL reference cards” containing join recipes, filter templates, example CTEs and verbatim SQL fragments; about 60% of each card’s token budget is concrete SQL examples.

Content moved more than the harness

The clean comparison holds the generation harness fixed and replaces handcrafted table documentation with optimized cards. On an internal held-out sample of 517 queries, Claude Sonnet 4.6’s AST similarity rose from 0.490 to 0.550, a roughly 12% relative gain. Qwen Coder 3-30B rose from 0.407 to 0.509, about 25%.

Harness optimization—changing retrieval tools and the prompt while keeping the baseline knowledge base—produced smaller gains in the same evaluation: Sonnet moved from 0.490 to 0.503, roughly 3%, while Qwen moved from 0.407 to 0.456, roughly 12%. Combining the optimized harness with optimized cards reached 0.570 for Sonnet and 0.519 for Qwen, but the paper says the two surfaces partly substitute for one another once better evidence is available.

Execution results on a smaller internal cohort point the same way but are not decisive. With 102 executable queries, cards alone moved Sonnet execution accuracy from 0.255 to 0.333 and Qwen from 0.176 to 0.235. The paper says all execution-accuracy confidence intervals overlap within each model block, so those numbers are directional rather than statistically resolved.

The public check is a warning label

The authors also test a fixed 300-question subset of BEAVER’s dw development split using Claude Sonnet 4.5 and official set-based execution scoring. The best configuration—cards plus retrieved raw SQL—scores 9.00%, versus 6.33% for the team’s few-shot reproduction. That 2.67-point difference has a paired-test p-value of 0.12.

Cards alone score 6.67%, nearly tied with raw-query retrieval at 6.33%. The combined arm also receives more total context. The public result therefore does not establish that distilled cards beat raw examples; it only shows the combined context directionally leading on this subset.

The practical rule is narrower than “stop tuning agents.” Before adding another retrieval call or prompt mutation, measure whether the retrieved knowledge itself contains the joins, filters and intermediate structures users repeatedly need. In this study, improving that reusable evidence bought more than harness tuning on the production structural benchmark. The public execution test says not to assume the advantage transfers unchanged to a benchmark without real usage histories.