AgentOnto’s 87.5% result tests when an analytics agent should refuse—not just whether its SQL runs

AgentOnto reports an 87.5% domain-average analysis accuracy across 606 enterprise-analytics requests, compared with 55.0% for DIN-SQL, the strongest baseline by that average. The important part of the result is not only the size of the lead. It is what the paper counts as a correct answer: the system must choose the right action—execute, ask for clarification or declare the request unsupported—and, when it executes, return the correct result.

A broader target than execution accuracy

The evaluation contains 282 World Development Indicators (WDI) requests and 324 U.S. EPA Air Quality System (AQS) requests. AgentOnto handles 223/282 WDI requests (79.1%) and 311/324 AQS requests (96.0%) correctly. DIN-SQL reaches 44.3% and 65.7% on those domains; ReFoRCE-v8 reaches 43.3% and 63.3%; MAC-SQL reaches 44.3% and 43.8%; and Vanna reaches 46.5% and 40.4%.

Those are not public-leaderboard BIRD or Spider scores, and they should not be compared with them. The paper’s authors prepare a database, business documents and example analysis tasks for each of the two public data sources, then evaluate a private request set spanning executable and non-executable cases. In WDI, 206 requests should execute, 30 require clarification and 46 are unsupported; AQS contains 250 executable, 28 clarification and 46 unsupported requests.

That composition changes what “coverage” means. AgentOnto’s analysis module executes 75.7% of requests on average and records 95.9% analysis accuracy with a 5.5% silent semantic-error rate. DIN-SQL executes 98.4% but records 55.0% accuracy and a 44.0% silent-error rate. ReFoRCE-v8 executes 99.7% with 53.3% accuracy and 46.5% silent error. In this evaluation, answering almost everything is a liability because roughly one quarter of requests should not immediately produce a query result.

The strongest ablation is also the practical lesson

Giving the same ontology directly to a model is not enough. “Direct Ontology” averages 51.4% analysis accuracy and 50.9% silent error. AgentOnto’s staged ontology-guided analysis averages 95.9% and 5.5%, respectively. The claimed gain therefore comes from using the ontology to form and check a plan and connect it to database mappings—not merely inserting semantic context into a prompt.

The updating experiment is smaller. On 22 AQS feedback events and 29 requests in each of three behavioral categories, AgentOnto reports 100% update, correction, behavior-preservation and generalization accuracy. Context-only and memory/reflection baselines reach 89.7% correction and generalization, while behavior preservation falls to 89.3% and 86.2%.

What readers cannot yet reproduce

The paper says all generative methods use gpt-5.6-terra, with separate gpt-5.6-sol calls for reference construction and independent review. That controls the model within the authors’ comparison, but the paper does not link a downloadable copy of the 606 requests, the 44 total feedback events, reference answers or runnable code. It also reports ontology updating only on 22 AQS feedback events, not the full 44-event benchmark inventory.

So the defensible conclusion is narrower than “ontologies solve enterprise analytics.” AgentOnto shows, on two author-constructed workloads, that evaluating execution, clarification and refusal together can expose errors that SQL-only coverage hides. The 32.5-point average lead is notable; an artifact release is what would make it independently testable.