Block’s private benchmark is a warning against one-number production evals

Block’s latest internal benchmark does not test text-to-SQL. That limitation matters. But its design exposes a production-evaluation mistake that NL2SQL teams make constantly: reporting one aggregate score across workloads whose scale, drift and difficulty differ enough to change which system wins.

A benchmark built around deployment conditions

Block says BlockTabBench2026 samples more than 100 de-identified datasets from production machine-learning problems across its financial products. The datasets range from a few thousand to several million rows and from a few hundred columns into the mid-thousands. Each dataset uses an out-of-time split: training follows historical data, while testing uses unseen future data, matching the way Block says its models are developed and deployed.

That setup produces a more useful result than a single public-leaderboard rank. Block compared XGBoost, the TabM deep-learning approach and the in-context-learning TabFM foundation model, selecting winners by area under the precision-recall curve. Across datasets, XGBoost won 41%, TabM 37% and TabFM 22%.

The aggregate is not the real finding. Block reports that TabFM was strongest when labeled data was scarce, but lost its advantage as training volume grew. TabM overtook alternatives at larger scales. XGBoost’s wins clustered on problems where every method performed relatively well, while its advantage narrowed as the attainable ceiling fell.

The lesson for NL2SQL teams

Those percentages must not be transferred to SQL generation; Block did not evaluate NL2SQL. The transferable lesson is methodological: slice the evaluation by the conditions that alter system behavior.

For an NL2SQL deployment, that means publishing more than total execution accuracy. At minimum, split results by schema size, join depth, question frequency, availability of prior validated SQL, and whether the question arrives before or after a schema or business-definition change. Use time-based holdouts where production changes over time. Keep a separate slice for rare questions with few examples, because Block’s results show how the preferred modeling family can reverse when training evidence is scarce.

Block also chose PR-AUC because its production decisions operate at different precision-recall thresholds. NL2SQL teams need an analogous operating metric: not just whether a query executes, but the cost of a plausible wrong answer versus an explicit abstention at the threshold the business will actually use.

The practical test is simple. If a team cannot say which workload slice produced its headline score—or whether the winner changes across slices—it does not yet have a production benchmark. It has an average that may hide the deployment regime where its system fails.