dbt’s semantic-layer benchmark measures two interventions—not one

dbt Labs’ 2026 comparison gives the Semantic Layer a clear win over direct SQL generation. On its modeled project, Claude Sonnet 4.6 scored 98.2% through the Semantic Layer versus 90.0% with text-to-SQL; GPT-5.3 Codex scored 100.0% versus 84.1%. Those are useful results—but they do not isolate the effect of the semantic layer from the effect of restructuring the data underneath it. dbt’s report

What the test changed

The benchmark uses 11 questions over the ACME Insurance dataset, runs each question 20 times and compares four configurations: direct text-to-SQL, a minimal semantic layer, a modeled semantic layer, and text-to-SQL over the same modeled data. dbt says the original normalized schema required too many entity hops for MetricFlow to answer some questions, so an LLM created three additional dbt models that joined tables and exposed matching semantic models. The team then supplied those new tables and relationships to the direct-SQL agent too. dbt’s methodology and repository

That control is the most revealing part of the study. Before the three models were added, Sonnet’s direct-SQL accuracy was 64.5% across all 11 questions. On the modeled data it reached 90.0%—a 25.5-point gain without routing the question through MetricFlow. The Semantic Layer then raised the modeled result another 8.2 points to 98.2%. For GPT-5.3 Codex, the modeled comparison was 84.1% versus 100.0%. dbt’s report

The defensible conclusion is therefore narrower than “semantic layers beat text-to-SQL.” In this small vendor-run test, cleaner analytical models helped both approaches, while the semantic layer added determinism and a further accuracy gain on questions inside its scope.

Refusal is part of the product contract

The original-schema results also expose a difference that aggregate accuracy obscures. For questions requiring too many hops, the semantic layer scored zero because it could not express them; direct SQL sometimes answered them correctly. On questions within scope, both tested models reached 100% through the semantic layer. dbt’s operational argument is that an explicit refusal is safer than plausible wrong SQL for governed metrics. dbt’s report

That is a routing decision, not a universal replacement rule. A production agent can try the governed metric path first, label an out-of-scope refusal, and reserve read-only SQL for exploratory questions that a human can inspect.

How teams should reproduce it

The benchmark is open source, but its evidence remains narrow: one dataset, 11 questions, a vendor evaluating its own semantic layer, and an entire schema loaded into the direct-SQL context—an approach dbt itself says is impractical for larger datasets. dbt’s report and code

A local evaluation should separate three steps: raw schema, cleaned analytical models, and semantic-layer execution. Score correct answers, wrong answers and refusals separately. If modeling recovers most of the gain, fix the data model first; if governed execution eliminates the remaining silent errors, that is the semantic layer’s measurable contribution.