A data agent needs answerability tests—not just SQL execution tests

A SQL execution test can prove that a query ran. It cannot prove that an analytics agent found the right table, chose the authoritative dashboard, reconciled conflicting business definitions or recognized that the available data could not answer the question.

That distinction is the most useful lesson in Databricks’ engineering account of Genie. The company describes data-agent work as different from coding-agent work: code can often be checked against deterministic tests, while an enterprise analytics request may have no single executable specification. The relevant context can also span tables, notebooks, dashboards and documents, some of which may be stale or contradictory.

Do not turn one accuracy number into a deployment promise

Databricks reports that a combination of specialized knowledge search, parallel reasoning and a multi-model design raised Genie’s accuracy on an internal set of real-world analysis tasks from 32% to more than 90% relative to a leading coding-agent baseline. It also reports up to a 40% improvement on its table-discovery benchmarks from specialized search.

Those are vendor-reported results on internal benchmarks. The post does not publish the task count, task set or a reproducible evaluation package, so the percentages should be read as evidence for the architecture—not as an expected accuracy rate for another company’s warehouse.

The architecture still gives teams a practical testing plan. This is the desk’s derived checklist, not a Databricks product requirement:

1. Test asset discovery separately. Give the agent questions whose authoritative tables, dashboards and documents are known. Score whether it retrieves those assets before scoring the final answer.
2. Seed contradictions. Include two plausible sources with different definitions or reporting windows. Require the agent to identify the conflict and justify which source it treats as authoritative.
3. Include unanswerable questions. A production agent needs a scored path for “the available data is insufficient,” not only a reward for producing SQL.
4. Measure reconciliation, not merely execution. A query that runs can still encode the wrong metric or time window. Review intermediate evidence and calculations for a sample of tasks.
5. Record the cost of extra reasoning. Databricks says sampling multiple trajectories can improve accuracy but adds token and latency costs. Evaluate that tradeoff on your workload instead of enabling parallel reasoning universally.

The deployment gate

A useful acceptance suite should therefore have at least four scorecards: retrieval, authority selection, answer correctness and abstention. Latency and model cost belong beside them, because a technique that improves difficult investigations may be unnecessary for simple lookups.

The key shift is conceptual. Treat the generated SQL as one intermediate artifact in a larger investigation. If the evaluation starts and ends with “did the query execute?”, it misses the failures that data agents are specifically built to solve.