A regex bug manufactured an NL2SQL model gap

A new enterprise NL2SQL evaluation offers a useful warning: the code between a model and the database can distort a leaderboard as much as the model itself. In the DRL paper’s 1,000-pair PostgreSQL verification suite, an over-broad regular expression corrupted valid joins and created an apparent 4–10 percentage-point gap between vendors. After the authors fixed the expression and reran all three systems, GPT-4o, Claude Sonnet 4.5 and Gemini 2.5 Flash landed within roughly one point of one another. (DRL paper)

The benchmark broke correct SQL

The post-processor was intended to remove one of six known schema-owner prefixes from generated SQL. Instead, its pattern matched any identifier.identifier token. That meant a correct join such as c.customer_id = o.customer_id could become customer_id = customer_id. PostgreSQL then rejected the rewritten query as ambiguous, and the harness counted the failure against the model even though the generated SQL had been valid before sanitization. (DRL paper, Section 5.6)

The effect was large. On a Tier 1 sample, the faulty post-processor held the executed rate to 56–60% across the three model families. With the narrower expression, executed rates reached 98.2% for GPT-4o, 98.2% for Gemini 2.5 Flash and 93.1% for Claude Sonnet 4.5. Executed rate is not execution-match accuracy: it measures whether SQL ran, while execution match tests whether its result agreed with the gold query. The distinction is exactly why the bug was dangerous—it prevented otherwise scoreable queries from reaching the correctness test. (DRL paper, Sections 5.6–5.7)

The corrected result is a tie, not a winner

On the full corrected 1,000-pair suite, the reported PostgreSQL execution-match scores are 52.9% for GPT-4o, 52.8% for Claude Sonnet 4.5 and 52.1% for Gemini 2.5 Flash under schema-linked prompting. The paper reports heavily overlapping 95% Wilson intervals, so it treats the systems as statistically indistinguishable on this suite rather than ranking them by tenths of a point. (DRL paper)

The correction also changed the failure diagnosis. For GPT-4o’s 471 execution-match failures, semantic or filter errors accounted for 254, ordering or pagination for 68, projection mismatch for 64, invalid column references for 47, and generic execution errors for 38. The earlier harness had hidden some of those residual errors behind failures caused by its own rewrite. (DRL paper, Section 5.8)

The practical lesson is narrower than “benchmarks are buggy” and more actionable: sanitizers, prefix strippers and SQL repair steps are part of the measurement instrument. They need versioning, regression tests with known-correct joins, and before-versus-after artifacts. The DRL authors retain both pre-correction and corrected result sets in their artifact, making this particular failure auditable rather than merely disclosed. (DRL paper, Artifact Availability)