live wire
▸Snowflake’s Cortex Agent quickstart routes metrics and transcripts with prompt rules—but documents no authorization boundarySnowflake▸Oracle Select AI generates SQL/PGQ for property graphs—but errors if one AI profile mixes graphs with tables or viewsOracle▸Domo’s data team standardizes MCP builds with shared scaffolding; beta toolkits inherit user permissionsDomo▸Databricks proposes five outcome signals for agentic BI—and says dashboard counts and license usage miss decision qualityDatabricks▸One vibe-coded database app needed 56+ human test hours and three repair rounds despite AI Studio’s verification claimsMancas & Mancas / arXiv▸BigQuery AI.IF’s default cost mode trains a local model for most rows—and drops row-error thresholdsGoogle Cloud▸AWS’s multi-tenant data-agent pattern gives the broker Lambda no data permissions; Cognito roles carry Lake Formation scopeAWS Big Data Blog▸Amazon OpenSearch’s optimized engine adds nested agent traces, but AWS reserves it for log analytics—not vector searchAWS▸Two interpretation-based provenance semantics cannot express SQL bag difference, new theorem showsKolaitis / arXiv▸Databricks Feature Views reuse one definition for training and live scoring, but online lookup still returns latest available stateDatabricks▸Databricks Genie’s Slack and Teams previews default channel answers to public, even when viewers lack source-data accessDatabricks▸Snowflake’s two-tool GraphRAG agent beat a seven-tool graph setup 70% to 60% on 22 ontology questionsSnowflake Engineering▸Semantix opens an Apache-2.0 semantic-first data-agent stack, but labels the project WIP and has no release or adoption yetGitHub / icosmos-space▸Confluent defines Streamhouse as shared current state for agents and apps, but says broader semantics and governance are still to comeConfluent▸Snowflake’s Cortex Agent quickstart routes metrics and transcripts with prompt rules—but documents no authorization boundarySnowflake▸Oracle Select AI generates SQL/PGQ for property graphs—but errors if one AI profile mixes graphs with tables or viewsOracle▸Domo’s data team standardizes MCP builds with shared scaffolding; beta toolkits inherit user permissionsDomo▸Databricks proposes five outcome signals for agentic BI—and says dashboard counts and license usage miss decision qualityDatabricks▸One vibe-coded database app needed 56+ human test hours and three repair rounds despite AI Studio’s verification claimsMancas & Mancas / arXiv▸BigQuery AI.IF’s default cost mode trains a local model for most rows—and drops row-error thresholdsGoogle Cloud▸AWS’s multi-tenant data-agent pattern gives the broker Lambda no data permissions; Cognito roles carry Lake Formation scopeAWS Big Data Blog▸Amazon OpenSearch’s optimized engine adds nested agent traces, but AWS reserves it for log analytics—not vector searchAWS▸Two interpretation-based provenance semantics cannot express SQL bag difference, new theorem showsKolaitis / arXiv▸Databricks Feature Views reuse one definition for training and live scoring, but online lookup still returns latest available stateDatabricks▸Databricks Genie’s Slack and Teams previews default channel answers to public, even when viewers lack source-data accessDatabricks▸Snowflake’s two-tool GraphRAG agent beat a seven-tool graph setup 70% to 60% on 22 ontology questionsSnowflake Engineering▸Semantix opens an Apache-2.0 semantic-first data-agent stack, but labels the project WIP and has no release or adoption yetGitHub / icosmos-space▸Confluent defines Streamhouse as shared current state for agents and apps, but says broader semantics and governance are still to comeConfluent
nl2sql.ai
benchmarkBenchmark audit

GenDB-V makes generated query engines fast—but the real benchmark result is 16 wrong-answer baselines

The paper reports up to 61.3× geometric-mean speedups with machine-checked code, while showing why passing synthesis tests is not enough.

By The Benchmark Desk· Nov 4, 2026the quick take — two AI hosts go live when you do

GenDB-V’s largest number is a 61.3× geometric-mean speedup over DuckDB, but its most important result may be a correctness audit: the authors found valid inputs that produced wrong answers in 16 of 28 implementations generated by a testing-based baseline.

What the benchmark actually measures

The paper evaluates 22 TPC-H query templates at scale factor 100 and six SEC-EDGAR templates at scale factor 3. Each parameterized template uses three bindings. Runs cover one, 32 and 64 threads on one server with two 16-core Intel Xeon Gold 5218 processors and 384 GB of RAM. The timing protocol is one cold run followed by three hot runs, with the median hot time reported; template times are averaged across the three bindings.

Under joint storage-and-query optimization, GenDB-V reports geometric-mean speedups of 51.6× over DuckDB 1.5.6 and 22.0× over Umbra 26.08 on TPC-H SF100. On SEC-EDGAR SF3, the corresponding figures are 61.3× and 28.3×. Those are template-specialization results, not a drop-in general SQL-engine score.

The paper separately evaluates one generated general-purpose engine over the SQL fragment supported by its SQLSpec formalization. On the 22-query TPC-H SF100 workload, version 16 completes in 109.3 seconds at one thread and 9.16 seconds at 64 threads. That is 3.4× and 1.8× faster than DuckDB, and 1.52× and 1.27× faster than Umbra, at those respective thread counts.

Why the correctness result matters

GenDB-V does not ask the agent to define correctness. A deterministic translator constructs a fixed SQLSpec target, and Verus checks the generated Rust implementation and proof against it. The claimed guarantees cover SQL correctness within the declared fragment, data preservation, memory and arithmetic safety, race-free parallel execution and termination.

For comparison, the authors ran the testing-based Bespoke OLAP system through its maintained SynnoDB implementation with the same Claude Opus 5.5 setting. They then constructed valid databases that violated assumptions embedded in the generated code. Wrong answers appeared in 10 of 22 TPC-H templates and all six SEC-EDGAR templates. Examples included 32-bit arithmetic overflow, a hidden 64-line-per-order limit and assumptions that nation names are unique. One SEC-EDGAR failure also appeared on the original SF3 data for an unseen currency binding.

The boundary on the claim

This remains an author-run evaluation on one dual-socket machine, two analytical workloads and a deliberately bounded SQL fragment. TPC-H is also familiar training material, which is why the less familiar SEC-EDGAR workload is useful but not a substitute for independent reproduction. Synthesis was not free: the paper reports about $2,310 for joint template synthesis across both benchmarks and $1,229 for the general engine.

The repository publishes the system, benchmark folders, result packages and verified crates under a PolyForm Noncommercial license. The next decisive benchmark is therefore reproducibility: whether another team can regenerate the proofs and speedups on different hardware, then extend SQLSpec without weakening the fixed correctness target.

sources

Filed by The Benchmark Desk. Corrections: desk@nl2sql.ai · Our standards →

comments · 0

    Comments are moderated before they appear. Your email is used once to confirm it is you — never shown, never sold. Corrections and questions get an answer from the desk when we have one.

    more from the desk