Databricks says generated SQL is the easy part—validation is now the analytics bottleneck

Databricks has published a practical warning for teams rolling conversational analytics into production: making a query is no longer the scarce skill. Deciding whether the answer is fit for a business decision is. Its new AI-era analysis guide puts question framing and answer validation at opposite ends of an increasingly automated workflow.

The failure mode is a polished answer to the wrong question

The guide’s strongest point is operational, not philosophical. A request such as “why is revenue down?” leaves the agent to choose a revenue definition, customer population, calendar and comparison period. A syntactically valid query can therefore return a clean chart while silently answering a different question.

Databricks recommends treating the prompt as an analytical specification: name the decision, define the terms, set the population and time window, choose a baseline and state what result would change the plan. It also separates descriptive, diagnostic, forecasting and causal questions, warning that an agent will often choose the easiest calculation rather than the method needed to support the claim.

Seven checks before anyone acts

The proposed validation sequence is concrete:

1. Restate the question and definitions.
2. Confirm the source is authoritative, current and permitted for the user.
3. Inspect filters, joins, aggregation grain, row counts, nulls and duplicates.
4. Reconcile important totals to a certified metric or known-good report.
5. Break results apart by major segments.
6. Test plausible alternative methods or explanations.
7. Preserve the query, definitions, assumptions and limitations.

That fourth step is the dividing line between merely exposing generated SQL and operating a trustworthy analytics agent. SQL visibility makes an answer inspectable; reconciliation against a governed reference makes it testable.

Databricks illustrates the context problem with its own 13-question Genie walkthrough: raw tables produced no correct answers; descriptions, keys and known values raised accuracy to 54%; example queries took it to 77%; and written business rules reached 100%. Those are vendor-authored, small-sample results—not an independent product benchmark—but they isolate a useful deployment lesson: the model did not change while the business context did.

What teams should implement

Before expanding access, create a representative question set with known-good answers and rerun it across business domains and user permissions. Require high-impact answers to retain their generated query, metric definition, source lineage and owner. Most importantly, do not treat the model’s own confidence statement as validation; Databricks explicitly says that does not count.

The broader benchmark context supports the caution. Spider 2.0 was designed around enterprise workflows and large schemas rather than tidy academic databases. Databricks cites the original paper’s much lower task success on those realistic environments to argue that enterprise context—not SQL syntax—is the hard part.

For practitioners, the takeaway is straightforward: conversational analytics needs a review contract, not just a chat box. The minimum viable contract is certified definitions, visible query logic, reconciliation to known-good numbers and a reproducible evidence trail.