Spider 2.0-Lite’s pinned evaluator has a 62.16% ceiling before a model runs

A reproducibility audit merged into SiEval on September 4 identifies a structural problem in the pinned Spider 2.0-Lite evaluation path: the evaluator cannot execute the benchmark’s Snowflake cases, yet it still reports a score over the full 547-question set.

The arithmetic is visible in the primary sources. The pinned Spider 2.0-Lite data contains 547 questions: 205 BigQuery, 207 Snowflake, and 135 local SQLite cases, counted from instance_id prefixes in the benchmark JSONL. But the corresponding evaluate.py dispatches only identifiers beginning with bq or ga to BigQuery and local to SQLite. Every other prefix receives Unsupported instance id prefix. At the end, the script nevertheless prints correct_examples / 547 as its “Real score.”

That leaves only 340 of 547 cases reachable through the pinned evaluator. Even a hypothetical system correct on every routable case would report 62.16%. This is not a model-quality ceiling; it is a harness ceiling.

Why the denominator matters

Spider 2.0-Lite’s README describes a text-in/text-out task spanning local SQLite, BigQuery and Snowflake, and explicitly directs users to obtain Snowflake access. The data and gold outputs therefore describe a three-engine benchmark. The pinned scorer, however, implements two execution paths, not three.

SiEval’s newly merged port makes that mismatch explicit rather than papering it over. Its default run selects the 135 SQLite questions, reports that denominator, and publishes per-backend counts. The PR also marks the task experimental and warns that a local-only score is not comparable with an aggregate over all 547 questions. Its Snowflake execution path is first-party code, not a byte-for-byte reproduction of the upstream Lite evaluator.

The port reports one illustrative local-only run at 48.89% execution accuracy, 66/135, with zero pipeline failures and three execution errors. That figure should remain exactly what the denominator says: a result on the SQLite subset, not a Spider 2.0-Lite leaderboard result and not a basis for extrapolating performance on the cloud warehouses.

What to record in a result

Any Spider 2.0-Lite number should carry four fields beside it: evaluator revision, requested engines, attempted-question count, and per-engine coverage. Without those, two identical percentages can represent materially different tests.

The immediate lesson is narrower than “the benchmark is broken.” The benchmark corpus includes all three engines; one pinned public evaluator revision does not. Reproducibility therefore depends on naming the harness as precisely as the dataset.