BigQuery’s agent telemetry shows what accuracy-only SQL leaderboards leave out

Static SQL benchmarks answer an indispensable question: did the system produce the right result? BigQuery’s new agent-observability surface adds a different set of measurements drawn from production use. Google’s documentation says teams can monitor performance, adoption, latency and costs, including agent count, user count, conversation count, frequently used knowledge sources, projected token usage and hourly answer latency.

Those fields should not replace execution accuracy. They should become the second layer of an evaluation scorecard.

Keep correctness as the admission gate

The current BIRD leaderboard labels separate development and test execution-accuracy columns. Its newest high-ranking submission, RAS, is dated August 19, 2026 and reports 72.49% on development and 79.82% on test. Those are split-specific results; they should not be collapsed into one number.

Spider 2.0 illustrates another boundary. It publishes separate tables for Snow, DBT and Lite, which are different task settings. Their current leaders report 96.70 on Snow, 65.6 on DBT and 76.23 on Lite. Those scores are useful within their own tables, not as interchangeable measures.

A production scorecard should retain the same discipline. Start with a fixed, versioned prompt set and record result correctness for every run. Only systems that clear the correctness threshold should advance to latency or cost comparison. Otherwise, a fast wrong answer can look like an optimization.

Turn telemetry into a repeatable evaluation

BigQuery’s monitoring documentation exposes enough operational fields to build a four-part report:

1. Correctness: execution or result-set agreement on a versioned test set, reported separately for each split and task setting.
2. Latency: median and tail response time, using hourly answer latency to detect load-dependent regressions.
3. Cost proxy: projected token usage per accepted answer, not merely per conversation.
4. Adoption: active users, conversations and knowledge-source usage, segmented so heavy use is not mistaken for high quality.

The denominator matters. “Tokens per conversation” rewards short conversations even when users abandon them. “Tokens per accepted correct answer” couples efficiency to an outcome. Likewise, user engagement is a product signal, not proof of SQL correctness.

The useful gap

BIRD and Spider 2.0 remain controlled tests; BigQuery observability measures live behavior. The two surfaces answer different questions, so neither should absorb the other. A defensible release gate is sequential: pass the fixed correctness suite, then compare latency and cost, then watch production adoption for drift.

That structure also makes regressions diagnosable. If accuracy holds while hourly latency rises, investigate serving. If tokens rise while knowledge-source usage shifts, inspect grounding. If adoption rises while accepted-answer correctness falls, stop celebrating the traffic and repair the agent.