Data-agent incidents need two queues: answer quality and infrastructure health

Production data agents can fail while every conventional service indicator stays green. A model can choose the wrong tool, misunderstand a request or route work to the wrong specialist even though calls complete without errors. Conversely, an identity-policy change or regional throttle can produce degraded answers without surfacing a clean infrastructure exception. AWS’s new four-agent demonstration treats those as separate incident classes rather than forcing both through one dashboard. AWS describes the architecture and implementation here.

Split the first diagnostic question

The design pairs Amazon Bedrock AgentCore Evaluations with AWS DevOps Agent. AgentCore Evaluations samples production interactions and scores dimensions including helpfulness, correctness and goal completion; AWS says the evaluator can inspect conversation context, tools and task requirements, then group low-scoring sessions into recurring patterns. AWS DevOps Agent works the infrastructure side, correlating CloudWatch logs, metrics, traces and IAM state to investigate service failures and recommend remediation.

That split matters for text-to-SQL and conversational analytics. A syntactically valid query that answers the wrong business question is a quality incident. A query that fails because the agent lost warehouse permission is an infrastructure incident. The user may see the same symptom—no useful answer—but the operator should not send both cases through the same playbook.

AWS demonstrates the pattern on an airline-reservation swarm with a supervisor plus flight, user and reservation agents. The agents share working memory and can hand work directly to peers, so there is no fixed call graph. AgentCore runtime emits OpenTelemetry data into CloudWatch; evaluation results land there too. Infrastructure investigations are submitted to AWS DevOps Agent through a signed webhook.

What to copy—and what not to claim

The reusable idea is the operating boundary: maintain a quality queue driven by task outcomes and a health queue driven by permissions, throttling and service behavior. Join them at the session and trace identifiers so an on-call engineer can determine whether a bad answer came from reasoning, tool selection or the execution substrate.

The post also supplies source code for the four-agent system, including CDK infrastructure, an evaluation dashboard and the DevOps Agent integration. That makes the architecture inspectable rather than purely conceptual.

But this is not evidence that the combined system improves accuracy or incident resolution time. AWS provides an implementation walkthrough and example failure modes, not a controlled comparison, false-positive rate, evaluator agreement study or measured mean time to recovery. A data team adopting the pattern should therefore set its own release gate: label a representative sample of analytics sessions, test whether quality scores predict real user-visible errors, inject permission and throttling faults, and verify that the two queues route incidents to the right owners.

The practical lesson is narrow but useful: “the agent returned a bad answer” is a symptom, not a diagnosis. Production analytics teams need telemetry that can separate answer quality from infrastructure health before automating remediation.