DiagDBench’s best agent scores 56.24—and usually stops before performance is restored

A harder target than naming the fault

DiagDBench evaluates agents against 454 expert-annotated held-out test cases spanning PostgreSQL and MySQL, three database schemas and three hardware configurations. Each case provides a live container in which the agent must inspect an anomaly, remediate it and submit a diagnostic report—not merely classify a root cause. The accompanying training set contains 6,940 synthesized cases, but the reported leaderboard numbers discussed here are for the separate 454-case test split.[^1]

The top result is 56.24 out of 100 for a ReAct agent backed by GPT-6 Astra. A Codex scaffold using the same model scores 52.44, a 3.80-point gap that isolates the agent loop more cleanly than comparisons that change both model and harness. Holding the ReAct scaffold, prompt and tools fixed, GPT-6 Astra beats the next-best backbone, Qwen3.8-Max, by 11.32 points.[^1]

What 56.24 actually measures

This is not a pure success rate. DiagDBench assigns 70% of each sample’s score to execution-grounded remediation and 30% to diagnostic-report quality. The remediation component checks whether the post-fix workload runs, whether an optimized workload remains semantically equivalent and how its median latency compares with the normal baseline. Each latency measurement uses one warm-up followed by five timed runs.[^1]

Two parts still depend on an LLM judge. GPT-6.1 Sol assesses workload equivalence using three binary judgments and majority vote; it also scores the diagnostic report in three rounds, with the median retained. That makes 56.24 a composite under this evaluation stack, not a directly observed percentage of incidents fully repaired.[^1]

The revealing failure mode

The paper’s error analysis excludes recoveries whose post-remediation latency lands within 0.8× to 1.2× of baseline, then classifies the remaining outcomes. Among failures or incomplete recoveries, residual slowdown accounts for 56.2% of the ReAct/GPT-6 Astra cases and 52.8% of the Codex/GPT-6 Astra cases. In other words, the strongest agents often identify and address the primary fault but fail to validate that end-to-end performance has actually returned.[^1]

That distinction matters for database-agent claims. A persuasive report can coexist with a slow or broken workload, and a remediation script can execute without restoring service quality. DiagDBench is valuable because it exposes both gaps in one task.

The result should still be treated as an initial paper-reported baseline. The benchmark incorporates private enterprise incident records in its construction pipeline, and semantic/report scoring relies on a named LLM judge. Independent reproduction, released artifacts and judge-sensitivity tests would materially strengthen the comparison. Until then, every citation of 56.24 should include the 454-case held-out split, the 70/30 composite, the model, the scaffold and the judge.[^1]

[^1]: Zhao et al., “DiagDBench: Evaluating Agentic LLMs on Interactive Database Diagnosis and Anomaly Remediation,” arXiv:2611.07339 (2026)