More retrieval made one graph-repair benchmark worse—and the agent did not know when to stop
Across six 80-case evaluation suites, a learned router spent fewer tokens than always-on retrieval; the result is a warning against treating tool access as automatically useful.
Retrieval is an intervention, not a free upgrade
A new graph-repair study provides a unusually clean measurement of when retrieval helps an agent—and when it actively sends the agent in the wrong direction. The authors evaluated two building-automation deployments, IBAL and Pritoni, using three predicate mixtures for each: evidential, terminological and representational. Each of the six suites contains 80 held-out cases, spread evenly across eight violation-injection budgets from 2 to 35 perturbations.
Every case has a programmatic reference repair. The benchmark scores the agent’s proposed graph edits against that reference with F1, reports whether the repaired graph satisfies its schema, and counts every repair-time token used for graph exploration, research, summarization and repair generation. That distinction matters: a graph can become schema-valid through the wrong factual edits.
Always-on retrieval failed a whole regime
Giving the repair agent access to a researcher improved median F1 on five suites. But on the IBAL representational suite, always-on access reduced median F1 by 0.126 versus no retrieval, with 67 of 80 cases getting worse. The effect was not a formatting artifact: the study used paired cases, 5,000 bootstrap resamples for confidence intervals and a violation-budget-stratified Wilcoxon test; the reported p-value for this suite was below 0.001.
The failure example is instructive. Without retrieval, the agent found an existing connection-point node using the graph’s naming convention. With retrieval, a researcher returned broader building descriptions; the agent followed those terms, failed to find the existing node and created a duplicate. Retrieval supplied plausible context while distracting from the decisive local evidence.
Cost widened the problem. The no-retrieval baseline already consumed a median 296,000 tokens on that suite. Always-on retrieval delivered −0.036 F1 per additional 100,000 tokens. Across all six suites, only 6 of 960 always-on/no-retrieval runs failed schema validation, so the central issue was not producing valid graphs—it was choosing the correct edits.
A small router beat an LLM judge
The authors trained a separate XGBoost T-learner for each deployment. Its inputs were only the predicates in the current violation cluster; one model predicted F1 with researcher access and another predicted F1 without it. Training cases were disjoint from evaluation.
Across the six held-out suites, this router used fewer tokens than always-on retrieval in all six, improved F1 over no retrieval in five and beat always-on retrieval in four. On the harmful IBAL representational suite it enabled research for only 37% of clusters, limiting the damage—although no retrieval still remained both cheaper and more accurate there.
A zero-shot LLM router was not selective: it enabled retrieval on 99–100% of clusters in every suite and generally consumed as many or slightly more tokens than the always-on policy. The benchmark’s practical lesson is narrower than “learn a router.” Tool availability itself needs evaluation against a no-tool control, on the deployment’s own data. An agent that can decide whether to call a tool may still call it almost every time.
sources
comments · 0