live wire
▸IBM makes watsonx Orchestrate AgentOps, custom LLM judging and Bedrock-agent discovery generally availableIBM▸SchemaGate 0.1.45 fixes broken Oracle ADB wallet connections and an OCI stack pinned 28 releases behindSchemaGate▸PDI’s Amazon Quick procurement agent grounds spend answers in vendor, category and contract contextAWS Business Intelligence Blog▸BigQuery’s ML.METRICS example returns 0.84 accuracy but 0.30 macro-F1 on the same 100-row classification queryGoogle Cloud BigQuery docs▸SchemaGate 0.1.44 auto-selects sentence embeddings, lifting bundled-schema retrieval from 90/98 to 93/98SchemaGate▸SchemaGate 0.1.43 adds read-only SQL execution with per-principal table checks—and documents unauthenticated client assertionsSchemaGate▸Databox adds reusable AI Analyst Skills with personal/company scope, auto-matching and marketplace installsDatabox▸Fabric previews an AI builder for data-agent instructions, source guidance and example queriesMicrosoft Fabric▸Databricks trains data-agent retriever to stop early or spend bounded extra search steps, reporting 5.8-second latencyDatabricks▸ThoughtSpot adds SpotterCode coding agent to its Visual Embed PlaygroundThoughtSpot▸LongMemEval-S audit: 67–73% of restore-fixable 80k-budget errors came from evicted evidence under three policiesarXiv▸SchemaGate 0.1.42 adds dimension-aware retrieval and fixes complex multi-table SQL promptsSchemaGate▸Snowflake agent toolsets can silently drop inherited tools when callers lack accessSnowflake Documentation▸Looker’s VS Code extension reaches GA with MCP-assisted LookML generation, editing and validationGoogle Cloud Looker release docs▸IBM makes watsonx Orchestrate AgentOps, custom LLM judging and Bedrock-agent discovery generally availableIBM▸SchemaGate 0.1.45 fixes broken Oracle ADB wallet connections and an OCI stack pinned 28 releases behindSchemaGate▸PDI’s Amazon Quick procurement agent grounds spend answers in vendor, category and contract contextAWS Business Intelligence Blog▸BigQuery’s ML.METRICS example returns 0.84 accuracy but 0.30 macro-F1 on the same 100-row classification queryGoogle Cloud BigQuery docs▸SchemaGate 0.1.44 auto-selects sentence embeddings, lifting bundled-schema retrieval from 90/98 to 93/98SchemaGate▸SchemaGate 0.1.43 adds read-only SQL execution with per-principal table checks—and documents unauthenticated client assertionsSchemaGate▸Databox adds reusable AI Analyst Skills with personal/company scope, auto-matching and marketplace installsDatabox▸Fabric previews an AI builder for data-agent instructions, source guidance and example queriesMicrosoft Fabric▸Databricks trains data-agent retriever to stop early or spend bounded extra search steps, reporting 5.8-second latencyDatabricks▸ThoughtSpot adds SpotterCode coding agent to its Visual Embed PlaygroundThoughtSpot▸LongMemEval-S audit: 67–73% of restore-fixable 80k-budget errors came from evicted evidence under three policiesarXiv▸SchemaGate 0.1.42 adds dimension-aware retrieval and fixes complex multi-table SQL promptsSchemaGate▸Snowflake agent toolsets can silently drop inherited tools when callers lack accessSnowflake Documentation▸Looker’s VS Code extension reaches GA with MCP-assisted LookML generation, editing and validationGoogle Cloud Looker release docs
nl2sql.ai
guideBENCHMARK LITERACY

Why BIRD’s 82.39 and Spider 2.0’s 96.70 are not comparable

A useful NL2SQL score needs a benchmark, setting or split, metric and evaluation state—not just a percentage.

Two benchmark scorecards show different NL2SQL scores and settings.
Side by side: what changed
By The Benchmark Desk· Sep 12, 2026the quick take — two AI hosts go live when you do

Two numbers currently dominate the NL2SQL leaderboards: 82.39 for DataGallery-Text2SQL on BIRD’s test split and 96.70 for Genloop’s Sentinel Agent v2 Pro on Spider 2.0-Snow. The larger number is not evidence that Spider 2.0 is easier, that Sentinel is better than DataGallery, or that either system would retain the same ordering elsewhere. They are coordinates in different evaluation spaces, not entries in one race.

Copy the benchmark and the setting

BIRD is a cross-domain benchmark with 12,751 question-SQL pairs, 95 databases and 37 professional domains. Its repository defines execution accuracy (EX) and a separate Valid Efficiency Score (VES). The live BIRD table currently gives DataGallery 78.10 dev EX and 82.39 test EX. Those two values belong to different splits; neither should be silently substituted for the other.

Spider 2.0 is not one undifferentiated test. Its repository documents three current settings: Spider 2.0-Snow, a 547-example text-to-SQL task entirely on Snowflake; Spider 2.0-Lite, another 547-example task split across 214 BigQuery, 198 Snowflake and 135 SQLite examples; and Spider 2.0-DBT, a 68-task DuckDB code-agent evaluation. The live leaderboard’s 96.70 leader is in the Snow setting. Calling that simply “96.70 on Spider 2.0” drops information needed to interpret the result.

Copy the metric and evaluation state

A percentage is incomplete without its metric. BIRD’s EX asks whether execution returns the expected result, while its repository treats VES as a separate efficiency-oriented evaluation. A report that says only “82.39” makes it impossible to know which behavior was measured.

Evaluation state matters too. Spider 2.0 says leaderboard results can change when evaluation examples are updated. It also requires submissions using provided oracle tables to identify that assistance, and its repository reported a disruption to Snow access on August 12, 2026. Those conditions do not invalidate the table; they are part of the test card a reader needs.

The minimum citation

For every NL2SQL result, copy at least four fields:

  1. Benchmark and version
  2. Setting or split
  3. Metric
  4. Evaluation date and assistance conditions

That turns “96.70 beats 82.39” into two defensible statements: Sentinel Agent v2 Pro leads Spider 2.0-Snow at 96.70 on the current table; DataGallery leads BIRD at 78.10 dev EX and 82.39 test EX on the current table. Anything stronger needs a controlled, same-setting comparison.

sources

Filed by The Benchmark Desk. Corrections: desk@nl2sql.ai · Our standards →

comments · 0

    Comments are moderated before they appear. Your email is used once to confirm it is you — never shown, never sold. Corrections and questions get an answer from the desk when we have one.

    more from the desk