DataKernelBench asks LLMs to optimize SQL workloads, not just write them

DataKernelBench moves the database-LLM question one layer below text-to-SQL: given a validated implementation of a query, can a model rewrite it into a faster GPU program? The benchmark, posted to arXiv on August 25 and accepted at EMNLP 2026, evaluates ten proprietary and open-weight models across CUDA and Triton on all 22 TPC-H queries at scale factor 10 using one NVIDIA H100. Its strongest configuration, GPT-5.5 with full-query CUDA optimization, reached 2.112× overall speedup over the compiled TorchPlan baseline with a 100% functional-correctness pass rate. (paper; leaderboard)

A benchmark with a fixed semantic contract

For each SQL query, the authors first generate a PyTorch program called a TorchPlan. In the main setup, Claude Opus 4.7 generated those plans; each was checked against DuckDB, and only validated plans entered the benchmark. The same fixed TorchPlans then served as both the correctness reference and the baseline compiled with torch.compile. An appendix repeats the experiment with GPT-5.5-generated TorchPlans, a useful control because the choice of intermediate program can change absolute runtimes. (paper, Sections 3.2 and Appendix D)

Evaluated models could optimize either the tensor-heavy core or the full internal query implementation while preserving the external API. They generated CUDA or Triton through an execution-guided repair loop. The published leaderboard reports suite-level speedup across 22 queries and falls back to compiled TorchPlan when a generated implementation is invalid or not fast enough; the headline number is therefore an end-to-end portfolio result, not a claim that every generated kernel individually ran 2.112× faster. (repository methodology; project page)

Correctness separates the leaders

Three proprietary-model configurations combined full correctness with more than 1.5× overall speedup: GPT-5.5 CUDA-full at 2.112×, Claude Sonnet 4.6 Triton-full at 1.543×, and Claude Opus 4.7 CUDA-full at 1.505×. Gemini 3.1 Pro Preview’s best compact-leaderboard row reached 1.443×, but passed only 54.5% of the 22 queries. (compact leaderboard)

The strongest open-weight row was Qwen3.5-397B-A17B-FP8 with Triton-full at 1.264× and 100% pass rate. GPT-OSS-120B with CUDA-full was effectively tied on speed at 1.263×, but its pass rate was 86.4%. DeepSeek-V4-Flash’s best row used Triton at the restricted core level, producing 1.233× overall speedup with a 90.9% pass rate. (compact leaderboard)

The practical contribution is not a new SQL-generation score. It is a reproducible test of whether coding models can preserve database semantics while specializing recurring analytical queries for GPU execution—and whether their gains survive accounting for failures across a complete query suite.