A data-agent evaluation program needs three loops—not one accuracy score

Enterprise data-agent teams are starting to use the word “evaluation” for three jobs that are related but not interchangeable. Recent guidance from Google, Databricks and Snowflake makes the split unusually clear: one loop reconstructs what the agent actually did, one turns user-reported failures into governed corrections, and one measures releases against a reproducible judge.

Loop 1: reconstruct execution

Google’s plugin team found that an evaluation harness could dispatch a skill-loading call and documentation searches in the same tool-call batch. Because the full skill instructions arrived only on a later turn, the searches had already run before the guidance entered context. A flattened event trace concealed that ordering bug. Google also says a final-answer judge marked three prompts in an 11-prompt suite as failures even though the raw trajectories showed that the required behavior had happened. Its recommendation is to preserve turn structure, inspect tool trajectories and compare live execution with plan-only runs. Google’s evaluation-engineering account documents both failures.

This is the debugging loop. It answers: did the harness, policy and tools execute in the intended order?

Loop 2: close production feedback

Databricks documents a separate operational loop for Genie Agents. User feedback does not automatically change agent behavior. A manager reviews the conversation and generated SQL, edits and reruns the query, then can save the corrected question-and-query pair as an instruction. Interactions can also become benchmark items, and Databricks warns that responses are non-deterministic. The Genie monitoring documentation describes the review, correction and benchmark workflow.

This is the maintenance loop. It answers: which real user failure should become a durable instruction or regression case, and who approved that change?

Loop 3: keep release scores comparable

Snowflake’s Cortex Agent evaluations add a third concern: the measuring instrument itself changes. Its v3 judges use Claude Sonnet 4.6 or GPT-5.4 with context windows above one million tokens, replacing earlier 200,000-token configurations. Snowflake explicitly says scores from different metric versions are not comparable. Teams can pin an exact minor version for reproducibility, but a run pinned to a deprecated version eventually fails rather than silently substituting a new judge. Snowflake’s metric-version documentation sets out those rules.

This is the release loop. It answers: did this agent version improve under the same dataset, judge and threshold?

Build three artifacts

Collapsing the loops creates predictable mistakes: teams rewrite prompts to fix harness races, treat thumbs-downs as automatic learning, or splice scores from different judges onto one trend line. A practical program keeps three artifacts instead:

1. Structured execution traces with turn boundaries, policy-loading points, tool arguments and side effects.
2. A reviewed correction ledger linking each production complaint to the accepted instruction or benchmark case.
3. A versioned release scorecard naming the dataset, agent build, judge version and promotion threshold.

The three loops should feed each other, but they should not share one pass/fail number. Execution evidence explains failures, production review decides what to change, and a pinned scorecard decides whether that change is safe to promote.