Snowflake’s 21-point text-to-SQL gain is a semantic-model result—not a leaderboard score

Snowflake has published a useful result for teams deciding where to invest in text-to-SQL: keep the model fixed, improve the semantic context, and accuracy can move substantially. In its experiment, Claude 3.5 Sonnet averaged 57% without a proper semantic model and 78% through Cortex Analyst, a 21-point increase across four BIRD-SQL development-set domains. That is strong evidence for semantic engineering—but it is not a new BIRD leaderboard score. Snowflake

What Snowflake actually tested

Snowflake used the same Claude 3.5 Sonnet model for its vanilla baseline and Cortex Analyst arm. The semantic-model system supplied business-facing names, metrics, filters, synonyms, table relationships and custom instructions. Specialized agents then helped create relationships, edit the semantic model and write global query instructions; Mistral Large 2 served as the LLM judge in a two-step evaluation that first compared columns and then compared result data frames when they differed. Snowflake

The reported evaluation covered 392 questions across four BIRD development domains: 60 debit-card questions, 80 California-schools questions, 111 thrombosis-prediction questions and 141 toxicology questions. The per-domain gains were uneven: 52% to 83%, 63% to 80%, 45% to 70%, and 69% to 79%, respectively. That variation matters because it shows the semantic layer did not contribute a universal fixed uplift. Snowflake

Why this is not a leaderboard comparison

Snowflake says it filtered BIRD-SQL, adapted MySQL examples to Snowflake SQL and removed examples that could not run directly. It also focused on four development-set domains rather than reporting a held-out BIRD test submission. Those choices are reasonable for product engineering, but they make the 78% figure specific to this experiment. Comparing it directly with a public leaderboard entry would mix different subsets, dialect handling and evaluation procedures. Snowflake

There is another caveat: the semantic-model agents were allowed to use correct SQL and question evidence while refining context. That makes this closer to building and validating a governed semantic asset from known examples than to measuring zero-shot performance on an unseen warehouse. Snowflake

The practical takeaway

The most defensible conclusion is architectural. Snowflake held the generator model constant and changed what the system knew about joins, metrics and business language. The largest domain gain—31 points on debit-card specialization—came from reducing ambiguity around relationships and terms; Snowflake’s worked example shows the ungrounded model joining transaction and product tables when the answer belonged in a gas-station segment field. Snowflake

Teams should reproduce that test on their own recurring questions: freeze the model, score a baseline, add reviewed relationships and definitions, then rerun the same set. The value of Snowflake’s report is not that it crowns a system. It demonstrates a controlled way to measure whether semantic-model work is buying accuracy in the domains that matter.