A new French open-data test finds architecture matters less once the local model reaches 3B parameters
OpenData Agent’s committed 120-question run shows specialists helping the smaller model, while a 3B single agent nearly closes the gap—and execution success still misses semantic SQL errors.
A small benchmark with a useful architecture comparison
OpenData Agent is a new MIT-licensed repository that combines two French open-data sources: a SQLite table of 1,161 French museums and 761 PIAF document passages. Its authors compare three designs on 120 questions—60 templated table questions with independently computed gold answers and 60 seeded PIAF questions: RAG-only, one tool-using agent, and a router that delegates to SQL or retrieval specialists. The project commits per-question traces and a generated results report rather than publishing only aggregate claims. Source: repository
The clearest result is not that “multi-agent wins.” With Qwen2.5-3B-Instruct, the single agent scores 0.717 overall and router-plus-specialists scores 0.725—a one-question-scale gap on this sample. The specialist design does use fewer tokens: 765.8 per question versus 897.7, about 15% less. Both reach 0.833 accuracy on the 60 table questions. Source: committed results
Specialization helps the smaller model
The architecture choice matters more with Qwen2.5-1.5B-Instruct. Router-plus-specialists reaches 0.633 overall and 0.833 on table questions, while the single-agent design reaches 0.525 overall and 0.600 on table questions. The repository’s error analysis says the weaker single agent often emits executable SQL that omits a needed aggregation, returning one row where the question requires a total. Source: repository
That distinction is operationally important. After one correction attempt, reported SQL execution-failure rates are at most 3.3% across the SQL-using runs, yet answer accuracy is materially lower. The residual failures are therefore largely semantic—missing aggregation or invented filters—not parser failures. Teams measuring only whether generated SQL executes would miss the harder class of mistakes. Source: committed results
RAG-only is also a useful negative baseline. It scores zero on every table-question run because passage retrieval cannot compute counts, sums or rankings over structured records. With the 3B model, adding a SQL-capable agent raises overall accuracy from 0.317 to 0.717–0.725. Source: committed results
What this does not establish
The repository is unusually candid about limits. The SQL half uses six templates, the two sources cover different topics—making routing easier—and differences below roughly five percentage points are noisy on 120 questions. Answer matching can also accept verbose responses containing the correct value, and only 1.5B and 3B local models were tested. There is no independent rerun yet. Source: repository
That makes OpenData Agent better treated as an inspectable architecture probe than a general leaderboard. Its strongest contribution is the reproducible comparison: specialization can rescue a weaker local model, but once the model is capable enough, a simpler single-agent design may deliver nearly the same correctness at a modest token premium.
sources
- OpenData Agent repository and methodologygithub.com
- OpenData Agent committed results reportgithub.com
comments · 0
- Databricks made ai_decide more accurate—but the model behind the SQL function can change againNov 17, 2026
- ThoughtSpot’s Data Cache is fast—but cached and live answers are not always equivalentNov 17, 2026
- Genie One scheduled tasks can now email colleagues—treat the recipient list as a data-delivery surfaceNov 17, 2026