live wire
▸Snowflake’s Cortex Agent quickstart routes metrics and transcripts with prompt rules—but documents no authorization boundarySnowflake▸Oracle Select AI generates SQL/PGQ for property graphs—but errors if one AI profile mixes graphs with tables or viewsOracle▸Domo’s data team standardizes MCP builds with shared scaffolding; beta toolkits inherit user permissionsDomo▸Databricks proposes five outcome signals for agentic BI—and says dashboard counts and license usage miss decision qualityDatabricks▸One vibe-coded database app needed 56+ human test hours and three repair rounds despite AI Studio’s verification claimsMancas & Mancas / arXiv▸BigQuery AI.IF’s default cost mode trains a local model for most rows—and drops row-error thresholdsGoogle Cloud▸AWS’s multi-tenant data-agent pattern gives the broker Lambda no data permissions; Cognito roles carry Lake Formation scopeAWS Big Data Blog▸Amazon OpenSearch’s optimized engine adds nested agent traces, but AWS reserves it for log analytics—not vector searchAWS▸Two interpretation-based provenance semantics cannot express SQL bag difference, new theorem showsKolaitis / arXiv▸Databricks Feature Views reuse one definition for training and live scoring, but online lookup still returns latest available stateDatabricks▸Databricks Genie’s Slack and Teams previews default channel answers to public, even when viewers lack source-data accessDatabricks▸Snowflake’s two-tool GraphRAG agent beat a seven-tool graph setup 70% to 60% on 22 ontology questionsSnowflake Engineering▸Semantix opens an Apache-2.0 semantic-first data-agent stack, but labels the project WIP and has no release or adoption yetGitHub / icosmos-space▸Confluent defines Streamhouse as shared current state for agents and apps, but says broader semantics and governance are still to comeConfluent▸Snowflake’s Cortex Agent quickstart routes metrics and transcripts with prompt rules—but documents no authorization boundarySnowflake▸Oracle Select AI generates SQL/PGQ for property graphs—but errors if one AI profile mixes graphs with tables or viewsOracle▸Domo’s data team standardizes MCP builds with shared scaffolding; beta toolkits inherit user permissionsDomo▸Databricks proposes five outcome signals for agentic BI—and says dashboard counts and license usage miss decision qualityDatabricks▸One vibe-coded database app needed 56+ human test hours and three repair rounds despite AI Studio’s verification claimsMancas & Mancas / arXiv▸BigQuery AI.IF’s default cost mode trains a local model for most rows—and drops row-error thresholdsGoogle Cloud▸AWS’s multi-tenant data-agent pattern gives the broker Lambda no data permissions; Cognito roles carry Lake Formation scopeAWS Big Data Blog▸Amazon OpenSearch’s optimized engine adds nested agent traces, but AWS reserves it for log analytics—not vector searchAWS▸Two interpretation-based provenance semantics cannot express SQL bag difference, new theorem showsKolaitis / arXiv▸Databricks Feature Views reuse one definition for training and live scoring, but online lookup still returns latest available stateDatabricks▸Databricks Genie’s Slack and Teams previews default channel answers to public, even when viewers lack source-data accessDatabricks▸Snowflake’s two-tool GraphRAG agent beat a seven-tool graph setup 70% to 60% on 22 ontology questionsSnowflake Engineering▸Semantix opens an Apache-2.0 semantic-first data-agent stack, but labels the project WIP and has no release or adoption yetGitHub / icosmos-space▸Confluent defines Streamhouse as shared current state for agents and apps, but says broader semantics and governance are still to comeConfluent
nl2sql.ai
benchmarkLeaderboard movement

BIRD’s new leader wins on execution accuracy—but not on efficiency

AskData’s 83.40% hidden-test result changes the main ranking, while a second update shows why Mini-Dev’s three SQL dialects should not be collapsed into one score.

By The Benchmark Desk· Oct 25, 2026the quick take — two AI hosts go live when you do

AskData moves to the top of BIRD’s hidden test

BIRD merged a new official AskData row on October 25, dated October 22, reporting 78.49% execution accuracy on the dev split and 83.40% on the hidden test split. That test result is 0.45 percentage points above GrainSQL’s 82.95%, making AskData the new leader on the benchmark’s primary execution-accuracy table. The update replaces AskData’s older 81.95% test row rather than leaving both submissions in place. BIRD’s merged leaderboard change

The same page puts human performance at 92.96% on the test split, leaving AskData 9.56 points below that reference. That comparison is test-to-test; the 78.49% dev number should not be used to compute the gap. BIRD leaderboard

The efficiency table tells a different story

AskData does not lead every BIRD metric. Its new test R-VES is 80.21, while GrainSQL remains ahead at 84.10 on the reward-based valid efficiency table. R-VES rewards correct queries while incorporating execution efficiency, so the two tables now have different leaders: AskData for execution accuracy, GrainSQL for efficiency-sensitive scoring. BIRD’s merged leaderboard change

That distinction matters because “first on BIRD” is no longer a complete description. The precise claim is that AskData leads the hidden-test execution-accuracy ranking as of October 25; it does not lead test R-VES.

Mini-Dev exposes a second axis: SQL dialect

A separate BIRD merge added SchemaGate with Claude Opus 5.5 to the 500-question BIRD Mini-Dev table. The system scored 81.00% execution accuracy on SQLite, 72.60% on PostgreSQL and 72.40% on MySQL. Those are three evaluations of the same 500 questions under different database engines, not interchangeable splits. BIRD Mini-Dev leaderboard commit

The submitted report also publishes a rerun: the SQLite score fell from 81.00% to 80.20%. It attributes one blank answer to an API error and notes that model outputs are nondeterministic. The result is self-reported, although prediction files, the runner and evaluator outputs are linked. SchemaGate Mini-Dev report

Together, the updates make the leaderboard harder to summarize with one number. Split, metric and dialect each change the conclusion: AskData leads hidden-test execution accuracy, GrainSQL leads hidden-test R-VES, and SchemaGate’s Mini-Dev result varies by 8.60 points between SQLite and PostgreSQL.

sources

Filed by The Benchmark Desk. Corrections: desk@nl2sql.ai · Our standards →

comments · 0

    Comments are moderated before they appear. Your email is used once to confirm it is you — never shown, never sold. Corrections and questions get an answer from the desk when we have one.

    more from the desk