Pinterest’s analytics agent improved compliance—but its evaluation exposed a 10% repeatability problem
The useful lesson is not the 89.3% score. It is the paired test design, case-level variance tracking and conservative acceptance gates behind it.
Pinterest’s latest production write-up offers a more useful blueprint for analytics-agent evaluation than another headline accuracy score. Its Analytics Agent searches internal tables, dashboards and metrics, inspects metadata, writes SQL and recommends answers. Pinterest found that getting those individual steps right requires testing not just whether the agent performs an action, but whether that action was actually needed. Pinterest Engineering published the method on November 13.
Test both action and restraint
For every compliance rule, Pinterest built two suites from labeled production chats. The required suite contains cases where the agent should take a step, such as reading a data-discovery skill or searching dashboards. The skip suite contains cases where that step is unnecessary. A change that pushed more dashboard searches fixed one required case but caused two new unnecessary searches; a one-sided test would have called that change a success. The team also kept a larger regression suite to protect unrelated behavior. The engineering post describes human-reviewed LLM labels, tuning and validation splits, plus a final untouched holdout.
That design matters for NL2SQL systems. “Always inspect every table” can raise a narrow compliance score while increasing latency and tool calls. “Always use the certified metric” can fail when no certified metric applies. Teams need negative examples that reward the agent for doing less.
Fresh sessions need structured memory
Pinterest first tried one persistent tuning session. It retained history, but the growing context became more expensive and less reliable. A memoryless loop cut estimated model inference cost by about fivefold, yet repeated failed approaches. The final design starts each iteration in a fresh session and passes a structured handoff: current scores, remaining failures, the best prior change, approaches not to repeat, suspected test defects and behaviors that must remain stable. Each iteration proposes up to four bounded changes, races them on a small shared sample, then evaluates the winner on the full tuning and validation suites. Pinterest’s account says edits were limited to prompts, skills, context and tools, with file and line limits shrinking over time.
A higher score alone could not accept a change. Pinterest also required it to repair more cases than it broke, pass a critic checking for hard-coded test answers, and preserve protected regression suites. Any one gate could reject the edit.
Treat repeatability as a first-class metric
The first large run reached a pooled 89.3% pass rate over tuning and validation cases, but most of the gain came from the first accepted edit and iterations 21 through 30 added nothing. More importantly, 10.3% of individual cases changed verdict when the same code was run twice. Similar aggregate totals concealed 23 case-level flips. Pinterest therefore tracks which exact cases move from pass to fail and does not trust a small-sample score increase by itself. Its footnotes say the internal evaluation used one trial per case and one base model and reasoning setting, so the result may not generalize across models.
The rollout lesson is straightforward: pair required and skip suites, keep a separate regression gate, record case-level flips, and reserve a holdout until the end. Pinterest says the process improved rule compliance, but it explicitly did not measure downstream answer quality. Compliance is a guardrail—not proof that an analytics answer is correct.
comments · 0
- Databricks made ai_decide more accurate—but the model behind the SQL function can change againNov 17, 2026
- A new French open-data test finds architecture matters less once the local model reaches 3B parametersNov 17, 2026
- ThoughtSpot’s Data Cache is fast—but cached and live answers are not always equivalentNov 17, 2026