Mnemiq’s benchmark says the biggest text-to-SQL gains came from plumbing, not a bigger model

Agentic Fabriq has released Mnemiq, an Apache-2.0 text-to-SQL engine designed to let teams swap models, retrieval depth, semantic enrichment and verification policy, then test those choices against their own databases. Its launch report includes results from BIRD mini-dev, Spider 1.0 and Spider 2.0-lite, plus a 28-configuration ablation study.

The headline comparison is eye-catching but needs the right label: this is a benchmark run by the maker of one of the systems being compared. Using its “got-facts” grader, Mnemiq reports 59.8% on Spider 2.0-lite, against 45.5% for Databricks Genie and 24.2% for Snowflake Cortex Analyst. It also reports case-flip rates of 2.2–4.1% across identical runs, versus 9.4–17.3% for Genie and 10.6–12.8% for Cortex Analyst. Agentic Fabriq publishes caveats: vendor models are undisclosed, each product used its own semantic setup, and the benchmark cannot isolate whether model choice, architecture or configuration caused the gaps.

The useful result is below the leaderboard

For practitioners, the ablation table matters more than the vendor ranking. Fixing a SQL-dialect mismatch improved one hosted-model configuration by about seven exact-match points—more than most model-level techniques. A local Qwen2.5-Coder 14B configuration on one RTX 4090 reached 50.9% exact match on BIRD mini-dev at roughly 3.7 seconds per question, essentially matching the best comparable hosted-model grid result of 50.7%. Moving from 14B to 32B did not improve exact-match accuracy in that setup.

More inference was not reliably better either. Five-sample self-consistency pushed one hosted configuration to roughly 34 seconds per question while scoring about seven points below asking the model once. The verifier was the only tested lever that consistently reduced wrong answers, although it did so by refusing more questions. On the 32B configuration, the reported wrong-answer rate fell to 25.7% while the system deferred on 28.2% of questions.

That split—right, wrong and refused—is a useful operating metric. An agent that protects its accuracy by declining everything is not useful; one that answers almost every question incorrectly is worse. Mnemiq exposes the threshold instead of collapsing those outcomes into one score.

An inspectable, not independent, test

The Mnemiq repository places authorization before schema retrieval, checks proposed SQL for read-only access, allowed objects, dialect compilation and EXPLAIN, and attaches a trace to each answer. Its six listed sources are Postgres, SQLite, DuckDB, Oracle, Snowflake and Databricks.

Grading lives in a separate Apache-2.0 project, Beacon. Beacon compares returned rows with stored gold results and keeps SQL as evidence rather than scoring SQL strings; its documentation says invalid runs are retained rather than deleted and per-question outputs can be inspected. Separation reduces the chance that the engine silently changes its own score, but Beacon and Mnemiq share the same vendor, so this is still not an independent benchmark.

The practical takeaway is narrower—and stronger—than “Mnemiq beats two vendors.” Teams should reproduce a small, tagged question set on their own warehouse, measure refusals and repeatability as well as accuracy, and fix dialect, authorization and retrieval boundaries before paying for a larger model.