Jeeves 1.1 catches valid SQL with the wrong answer shape—but exposes a harder join ceiling

A text-to-SQL system can generate syntactically valid, executable SQL and still answer a different question from the one the user asked. Jeeves 1.1 turns one version of that problem into an explicit post-execution check: when a question asks for one thing but execution returns a list, the Laravel package regenerates the query once with a prompt that states the expected shape.

The maintainer’s example is “Which carrier shipped the most orders?” A model can produce an otherwise valid ranking query without LIMIT 1; the first row is useful, but every additional row violates the singular request. A pre-execution SQL verifier cannot infer the mistake from validity alone. Jeeves therefore inspects the row count locally after execution, then sends the question, schema and a one-sentence shape correction for the retry. The release says no returned value, row or count is included in that prompt. The v1.1 release notes document the behavior and its privacy boundary.

Shape checks are narrow by design

This is a useful production pattern precisely because it is not presented as a general correctness oracle. Cardinality can contradict a question’s grammar, but a one-row result may still contain the wrong metric, filter or join. Teams adopting the pattern should treat answer shape as another deterministic assertion alongside SQL parsing, allowlists, read-only credentials and timeouts—not as proof that the answer is correct.

The release also adds optional semantic dataset matching before model-based routing. Exact routes retain priority; confident embedding matches can save the routing call, while misses, timeouts and unknown dataset names fall back to the existing path. The feature is off by default, and operators must supply both the embedding service and a corpus generated from their own schema descriptions.

The disclosed weakness matters more than the retry

Jeeves reports that the release does not improve its measured SQL accuracy: 35 of 46 questions, or 76%, on its own set, and 30–31 of 36 on a Spider development sample. The maintainer also breaks the internal set down by join width. Three-table questions scored 2 of 6, compared with 13 of 16 for two-table questions. The release says the full join path was already present in the prompt and that curating schema descriptions did not remove the failure.

Those figures are maintainer-reported results on small samples, not an independent benchmark, and the variable Spider score should not be read as a leaderboard result. Still, the disclosure gives evaluators a practical test plan: stratify questions by join width and answer cardinality, run the package’s benchmark command against the organization’s own schema and reference SQL, and inspect whether a retry fixed the semantics or merely changed the row count.

Jeeves 1.1’s most transferable lesson is that NL2SQL evaluation needs multiple failure labels. “Executed” is not the same as “answered”; “correct shape” is not the same as “correct query”; and aggregate accuracy can conceal a steep failure pocket in the joins a production schema actually requires.