An open Spider 2.0 audit says 4.2% of Snow’s test set is no longer reproducible

Spider 2.0-Snow’s published test setting contains 547 tasks. An open community audit now says 23 of those tasks—4.2% of the set—cannot be reproduced reliably against live Snowflake data and should be removed. The proposal has not been merged, so it does not change any official leaderboard score.

The pull request, filed May 5 and still open on August 30, would reduce the Snow test set from 547 to 524 instances. Its author attributes three removals to revoked Snowflake data shares and 20 to live-data drift: rolling windows, refreshed counts, regenerated identifiers and other source changes that make stored gold results disagree with current query output.

The examples show why a live-data benchmark needs versioning beyond a repository commit. Two tasks using NETHERLANDS_OPEN_MAP_DATA and one using AMAZON_VENDOR_ANALYTICS__SAMPLE_DATASET are reported to fail because their shares are no longer available. Other proposed removals include a GitHub-repository task whose top URL changes as packages are published, GA4 tasks with changing session counts, and an address task whose top 10 rows move as records are added.

This is not yet a maintainer-validated correction. The evidence and classifications are supplied by the contributor and its linked Spider2-E fork, not by an accepted upstream release. But the current upstream tree still contains the named gold files—for example the variants for sf006, sf008, sf009, sf012, sf013, sf029, sf037 and sf040—which confirms that the proposed removals are not part of the official evaluator.

The patch goes beyond deleting tasks. It reports reducing stored gold-result CSVs from 1,544 to 1,469, replacing one calendar-dependent gold SQL query with a version anchored to 2025, adding five accepted Snowflake-specific result variants, and changing the evaluator so an empty prediction can match an empty gold result. It also proposes default rejection rules for some hard-coded-answer SQL patterns. Those policy changes could affect scores independently of the 23-task denominator change and deserve separate review.

No reranking can be inferred from the patch alone: that requires every system’s task-level outputs to be rescored under the proposed 524-task evaluator. Until that happens—and until maintainers accept or reject the audit—the 547-task leaderboard remains the official record.

The practical consequence is narrower but immediate. Reproduction reports for Spider 2.0-Snow should name the repository commit, execution date, accessible Snowflake shares and evaluator policy. A single percentage without those four fields may describe a system, a moving dataset, or both.