live wire
▸Databricks makes Genie Code CLI GA, but disabling its web search does not stop shell-based internet accessDatabricks▸OpenData Agent finds 3B single-agent and router-specialist designs nearly tied on 120 French RAG+SQL questionsOpenData Agent repository▸Databricks says ai_decide is 33% more accurate, while its beta contract permits the underlying model to changeDatabricks▸Adaptive-Elbow reaches 97% table recall at 36.9% precision on merged BIRD Mini-Dev, but does not test SQL executionAdaptive-Elbow paper▸Databricks Genie One scheduled tasks can now email other users, including visualizations and a PDF of each runDatabricks▸ThoughtSpot Data Cache is GA for Snowflake and Databricks models, but cached answers can differ from live resultsThoughtSpot▸Tableau raises Cloud Manager ceiling to 200 sites, but Cloud+/Tableau+ base capacity still supports 50Tableau▸Databricks adds daily per-user Genie spending controls, but blocking remains approximate and uses list pricesDatabricks▸Databricks brings governed-tag propagation to Runtime 19+ classic compute, but ABAC can lag derived data by minutesDatabricks▸Snowflake adds Pegasus 1.5 video analysis to AI_COMPLETE at GA, while video processing remains public previewSnowflake▸Living Context Graph raises held-out BIRD transfer accuracy from 13.7% to 36.3%, but needs verified SQL outcomesLiving Context Graph paper▸Google previews Apache Iceberg views in BigQuery’s Lakehouse catalog, but query-time access still reaches every base tableGoogle Cloud▸Oracle makes OCI Agent Sandbox GA with managed Linux execution, project-scoped egress and 48-hour initial lifetimesOracle OCI▸IBM Cognos Analytics 12.2.0 makes Query Agent GA and expands its MCP server to 104 toolsIBM Cognos Analytics▸Databricks makes Genie Code CLI GA, but disabling its web search does not stop shell-based internet accessDatabricks▸OpenData Agent finds 3B single-agent and router-specialist designs nearly tied on 120 French RAG+SQL questionsOpenData Agent repository▸Databricks says ai_decide is 33% more accurate, while its beta contract permits the underlying model to changeDatabricks▸Adaptive-Elbow reaches 97% table recall at 36.9% precision on merged BIRD Mini-Dev, but does not test SQL executionAdaptive-Elbow paper▸Databricks Genie One scheduled tasks can now email other users, including visualizations and a PDF of each runDatabricks▸ThoughtSpot Data Cache is GA for Snowflake and Databricks models, but cached answers can differ from live resultsThoughtSpot▸Tableau raises Cloud Manager ceiling to 200 sites, but Cloud+/Tableau+ base capacity still supports 50Tableau▸Databricks adds daily per-user Genie spending controls, but blocking remains approximate and uses list pricesDatabricks▸Databricks brings governed-tag propagation to Runtime 19+ classic compute, but ABAC can lag derived data by minutesDatabricks▸Snowflake adds Pegasus 1.5 video analysis to AI_COMPLETE at GA, while video processing remains public previewSnowflake▸Living Context Graph raises held-out BIRD transfer accuracy from 13.7% to 36.3%, but needs verified SQL outcomesLiving Context Graph paper▸Google previews Apache Iceberg views in BigQuery’s Lakehouse catalog, but query-time access still reaches every base tableGoogle Cloud▸Oracle makes OCI Agent Sandbox GA with managed Linux execution, project-scoped egress and 48-hour initial lifetimesOracle OCI▸IBM Cognos Analytics 12.2.0 makes Query Agent GA and expands its MCP server to 104 toolsIBM Cognos Analytics
nl2sql.ai
guideProduction guide

Pinterest’s analytics agent improved compliance—but its evaluation exposed a 10% repeatability problem

The useful lesson is not the 89.3% score. It is the paired test design, case-level variance tracking and conservative acceptance gates behind it.

By The News Desk· Nov 14, 2026the quick take — two AI hosts go live when you do

Pinterest’s latest production write-up offers a more useful blueprint for analytics-agent evaluation than another headline accuracy score. Its Analytics Agent searches internal tables, dashboards and metrics, inspects metadata, writes SQL and recommends answers. Pinterest found that getting those individual steps right requires testing not just whether the agent performs an action, but whether that action was actually needed. Pinterest Engineering published the method on November 13.

Test both action and restraint

For every compliance rule, Pinterest built two suites from labeled production chats. The required suite contains cases where the agent should take a step, such as reading a data-discovery skill or searching dashboards. The skip suite contains cases where that step is unnecessary. A change that pushed more dashboard searches fixed one required case but caused two new unnecessary searches; a one-sided test would have called that change a success. The team also kept a larger regression suite to protect unrelated behavior. The engineering post describes human-reviewed LLM labels, tuning and validation splits, plus a final untouched holdout.

That design matters for NL2SQL systems. “Always inspect every table” can raise a narrow compliance score while increasing latency and tool calls. “Always use the certified metric” can fail when no certified metric applies. Teams need negative examples that reward the agent for doing less.

Fresh sessions need structured memory

Pinterest first tried one persistent tuning session. It retained history, but the growing context became more expensive and less reliable. A memoryless loop cut estimated model inference cost by about fivefold, yet repeated failed approaches. The final design starts each iteration in a fresh session and passes a structured handoff: current scores, remaining failures, the best prior change, approaches not to repeat, suspected test defects and behaviors that must remain stable. Each iteration proposes up to four bounded changes, races them on a small shared sample, then evaluates the winner on the full tuning and validation suites. Pinterest’s account says edits were limited to prompts, skills, context and tools, with file and line limits shrinking over time.

A higher score alone could not accept a change. Pinterest also required it to repair more cases than it broke, pass a critic checking for hard-coded test answers, and preserve protected regression suites. Any one gate could reject the edit.

Treat repeatability as a first-class metric

The first large run reached a pooled 89.3% pass rate over tuning and validation cases, but most of the gain came from the first accepted edit and iterations 21 through 30 added nothing. More importantly, 10.3% of individual cases changed verdict when the same code was run twice. Similar aggregate totals concealed 23 case-level flips. Pinterest therefore tracks which exact cases move from pass to fail and does not trust a small-sample score increase by itself. Its footnotes say the internal evaluation used one trial per case and one base model and reasoning setting, so the result may not generalize across models.

The rollout lesson is straightforward: pair required and skip suites, keep a separate regression gate, record case-level flips, and reserve a holdout until the end. Pinterest says the process improved rule compliance, but it explicitly did not measure downstream answer quality. Compliance is a guardrail—not proof that an analytics answer is correct.

Filed by The News Desk. Corrections: desk@nl2sql.ai · Our standards →

comments · 0

    Comments are moderated before they appear. Your email is used once to confirm it is you — never shown, never sold. Corrections and questions get an answer from the desk when we have one.