AWS makes the first failed turn—not the final answer—the unit of agent debugging

A data agent can produce four bad turns from one mistake. Counting four failures tells an engineering team how noisy the incident was; it does not tell them what to fix.

AWS’s newly published Agent Evaluation Metric (AEM) changes that unit of analysis. The method evaluates every response and tool-action turn, then labels later failures as prior_action_failed when they merely consumed an earlier bad output. In AWS’s illustrative five-turn sales-report conversation, the agent passes profit where revenue was required on turn two; three later turns inherit the damage, leaving one root cause rather than four independent defects. AWS describes the framework and example in its September 10 technical post.

What it adds to NL2SQL evaluation

This is a useful extension to answer-level testing, not a replacement for it. Microsoft Fabric’s preview evaluation SDK asks teams to supply natural-language questions and expected answers, then records summaries, per-question details and step-by-step execution data. It can also use a custom critic prompt for domain-specific equivalence judgments. Microsoft documents that workflow in its Fabric data-agent evaluation guide.

AEM adds causal structure to those traces. It splits correctness into truthfulness and completeness, checks whether a tool action chose the right tool and operation, and names specific defects such as missing parameters, extra parameters and inconsistent parameter values. The practical payoff is triage: when ten failing dialogs share an early tool_mismatch, teams can fix routing before tuning the prose of ten downstream answers. AWS’s failure taxonomy covers response, action, structural and cascade failures.

For text-to-SQL, the test fixture should therefore preserve the whole conversation: the clarification question, the selected data source, the business-metric parameter, the generated query and the answer. A wrong definition of “revenue” is not equivalent to a merely malformed final response, even if both produce a failed task.

Do not turn AEM into another magic number

AWS is explicit about the boundaries. The published implementation treats turn correctness as binary, uses semantic comparison rather than exact strings and presents 0.5 as a neutral starting threshold—not a tuned universal value. Its worked Amazon Quick section is illustrative and reports no internal production result. Correctness is also only the first dimension; AWS says safety, instruction retention and reasoning depth can be added later through the same decomposition pattern. Those limitations and extension points are stated in the launch post.

The rollout sequence is straightforward: keep the expected-answer benchmark, annotate the expected tool calls and parameters for multi-turn cases, mark order-invariant steps, then report both conversation success and first-root-cause distribution. Microsoft separately advises testing representative questions, inspecting generated queries and updating the context layer closest to the failure—schema selection, descriptions, source instructions or example queries. Its query-generation guidance maps those remediation paths.

The important shift is diagnostic. The final answer remains what users judge. The first wrong turn is where engineers should start.