Bolo’s model-remediation scores expose the gap between runnable and verified
An agent repaired most sampled inference pipelines, but the paper’s own hallucination filter removed nearly nine points of Type II coverage—and still needs human review.
A new model-remediation benchmark offers an unusually useful warning about what “working” means for agent-generated code. In the Bolo extended abstract, an agent turns model-weight repositories into inference pipelines and reaches 97.27% runtime-error-free coverage on Type II models and 86.08% on Type III models. But after Bolo applies its own code-hallucination filter, retained Type II coverage falls to 88.5%.
That gap is the result worth carrying forward. A pipeline that imports, loads and returns without an exception has passed an execution check; it has not necessarily shown that it loaded the intended model, consumed the supplied input or produced an output with the intended task semantics.
What the experiment actually tests
The paper evaluates three repository categories. Type I models are supported by the Hugging Face pipeline API but fail under standard use. Type II models use Transformers but lack a clear pipeline tag or sit outside supported pipeline tasks. Type III models are outside Transformers. The sample contains 5,444 Type I, 1,353 Type II and 1,581 Type III models, after excluding models larger than 16 GB and repositories with external service agreements.
For Type II and III, both Bolo and mini-swe-agent use ChatGPT-5.1-CodeX-mini, a 50-tool-call budget, and a 20-minute timeout per model. Bolo’s task-specific tools beat mini-swe-agent’s raw runtime coverage: 97.27% versus 93.05% for Type II, and 86.08% versus 73.12% for Type III. The cost comparison is less one-sided: Type II totals were $37.26 versus $36.65, while Type III cost Bolo $134.54 versus $53.60.
The verification penalty
Bolo’s HalluVer stage checks program structure and data flow: whether generated code loads the target model, computes on input and connects that computation to the output. It flagged 119 of 1,316 runnable Type II programs and 91 of 481 runnable Type I programs as hallucinated. Filtering reduced Type II coverage to 88.5%, and Type I coverage to 95.3% after Repair-2.1 and 95.6% after Repair-2.2.
The filter is not a final oracle. Human annotation found 48 valid Type II programs and 24 valid Type I programs among those flagged. Reported precision was 78.9% for Type II and 79.8% for Type I, despite recall of 100% and 98.5%, respectively. The authors therefore recommend human or LLM review before exclusion.
Benchmark takeaway
Bolo is preliminary—the paper is a five-page extended abstract—and HalluVer was evaluated on Type I and II, not Type III. Its strongest contribution is methodological: model-pipeline benchmarks should report at least three separate gates—runtime success, structural semantic checks and task-output validation. Collapsing those into one “coverage” score would overstate what an agent has actually verified.
sources
comments · 0