BigQuery’s predictive chat needs two scorecards: query routing and model quality

BigQuery’s conversational analytics preview can now turn a natural-language request such as “Predict the body mass of penguins” into a call to AI.PREDICT. That expands the product beyond retrieving or aggregating known values: the generated SQL can now invoke TabFM, Google’s pretrained tabular foundation model, to perform regression or classification. The change appeared in BigQuery’s September 8, 2026 release notes and is listed in the current conversational-analytics documentation.

The evaluation consequence is easy to miss. A single “correct answer” score now collapses two distinct systems: the agent that interprets the question and assembles the query, and the predictive model that estimates the label. Those layers can fail independently.

Score the orchestration first

The first scorecard should test whether the agent selected the intended data, label, features and task type. According to the AI.PREDICT contract, numeric labels trigger regression, while STRING and BOOL labels trigger classification. The prediction input must contain every feature present in the training input. The preview also caps inputs at 20 feature columns and classification at 10 categories.

That suggests a deterministic harness: hold out a set of natural-language prompts, inspect the generated SQL, and grade label selection, feature selection, filters and train/test separation before examining any prediction metric. A query can execute successfully while leaking the target, choosing the wrong time window or silently answering a classification question as regression.

Score the model separately

The second scorecard should evaluate TabFM against ground truth on a fixed holdout. Google exposes AI.EVALUATE for this purpose. For regression it returns mean absolute error, mean squared error, mean squared log error, median absolute error, R² and explained variance. For classification it returns macro-averaged precision, recall and F1 alongside accuracy.

That separation matters because the AI.PREDICT documentation’s penguin regression output is an example, not a general benchmark. Its displayed evaluation includes an R² of 0.9161 and mean absolute error of 158.34 for one randomly created 80/20 split. The query uses RAND() to assign rows, so readers should not treat those numbers as a stable product-wide accuracy claim.

Pin the test before rollout

A defensible acceptance test should therefore pin the prompt set, source snapshot, split seed or explicit holdout rows, generated SQL and prediction metrics. It should also report failures caused by unsupported width or class count rather than excluding them after the run.

Google has made predictive questions easier to ask; it has not made them reducible to text-to-SQL execution accuracy. For these workflows, “the SQL ran” is only the midpoint of evaluation.