Domain tools beat raw SQL in a 609-run MCP database benchmark

A new database-agent benchmark reports that small local models answered a fixed set of business questions more accurately when they selected domain-specific tools than when they generated SQL. The result is less an obituary for text-to-SQL than a sharp demonstration that interface design can dominate model size in bounded workflows.

The numbers

The MCP Blueprint paper compares three interfaces to the same PostgreSQL Sakila database: one execute_sql tool with six tables of DDL in the prompt; five domain-oriented tools backed by prewritten SQL; and a deliberately thin pack of generic, table-oriented tools. Four Ollama-served models from 3B to 8B attempted 17 customer-facing tasks three times at temperature zero. Of 612 planned model-by-interface-by-task runs, 609 completed.

Pooled across models, the domain-oriented pack scored 0.939, versus 0.666 for raw SQL and 0.605 for the generic pack. Fully correct runs were 174/204 (85%) for domain tools, 67/201 (33%) for raw SQL, and 63/204 (31%) for generic tools, according to the paper and the released benchmark repository.

The smallest tested model produced the largest jump. Llama 3.2 3B rose from 0.583 with raw SQL to 0.929 with the domain pack. Its tokens per correct answer fell from 31,476 to 2,723. Across all models, the domain pack averaged 3,056 tokens and 4.4 seconds per run, compared with 3,953 tokens and 17.3 seconds for raw SQL. These are fixed-hardware, task-specific measurements, not general price or latency estimates.

Why the generic pack matters

The useful control is not raw SQL alone. The generic pack also used parameterized tools, but scored below raw SQL overall. Its tools exposed table-shaped operations such as customer or film lookup, while the stronger pack exposed business operations such as account summaries and film recommendations. Joins, overdue logic and customer-standing rules lived in reviewed SQL behind those domain tools.

That contrast supports a narrower conclusion than “tools beat SQL”: task-aligned abstractions beat both unconstrained SQL generation and shallow wrappers in this setup. Simply adding tools constrained the model without necessarily supplying the semantics needed to answer the question.

Read the ceiling carefully

The authors disclose an important design limitation: the verticalized pack and rule-based scorer were developed against the same 17 tasks, and there was no held-out task set. The study covers one small, six-table sample schema; four local models; one decoding setting; and no frontier cloud models. The scorer used live database gold answers rather than an LLM judge, but included a fuzzy string threshold of 0.72.

So the benchmark does not establish that domain tools generalize to open-ended analytics. It does provide reproducible evidence for a practical production pattern: when questions recur and business rules are known, moving schema navigation and SQL synthesis behind tested domain operations can let a much smaller model deliver better measured outcomes. The next useful test is obvious—freeze the tool pack, then evaluate it on held-out requests and a larger enterprise schema.