Frozen BIRD predictions lose 4.4 points in a PostgreSQL replay

A pending BIRD pull request adds a result that is more useful than one more rank change. Ontology2SQL reports 70.20% execution accuracy on the public BIRD Mini-Dev SQLite split, then replays the same frozen model outputs through a deterministic PostgreSQL adapter and reports 65.80%. The 4.40-percentage-point difference puts a number on the portability loss that a SQLite-only score can hide.

The submitted Mini-Dev result covers 500 questions across 11 databases. On SQLite, the published breakdown is 127/148 simple questions correct (85.81%), 170/250 moderate (68.00%) and 54/102 challenging (52.94%), for 351/500 overall. The pipeline used DeepSeek V4 Flash, BIRD’s oracle evidence, reviewed database-local semantic mappings, one candidate per question, temperature zero and a 12-step initial agent budget.

The PostgreSQL result is not a second generation run. According to the project’s result report, an adapter parses each frozen SQLite prediction with sqlglot, applies PostgreSQL function semantics, resolves identifiers against the native catalog and runs read-only queries against the official combined PostgreSQL dump. That replay scores 123/148 simple (83.11%), 159/250 moderate (63.60%) and 47/102 challenging (46.08%), or 329/500 overall.

The portability gap widens with difficulty: 2.70 points on simple, 4.40 on moderate and 6.86 on challenging questions. That pattern does not prove that PostgreSQL itself makes semantic parsing harder—the adapter, stricter execution behavior and timeouts are all in the path—but it does show that a frozen set of nominally correct SQL predictions does not carry its full SQLite score across engines.

The artifact is unusually inspectable for a leaderboard proposal. The repository publishes both prediction files, packaged submissions and SHA-256 hashes, plus the adapter and evaluation entry points. It also discloses that 52 questions exceeded an initial 600-second wall clock and were recovered using only the first unbounded attempt; 20 still ended without a normal final answer and were scored wrong. For the PostgreSQL replay, 16 predictions did not execute: 11 were already empty in the SQLite artifact and five encountered PostgreSQL-specific strictness or timeout failures.

This is still a pending, self-reported Mini-Dev entry, not a merged hidden-test BIRD result. Its value is narrower and more practical: because the predictions are frozen and both engine-specific outputs are published, the 4.4-point gap can be inspected rather than attributed vaguely to “dialect differences.” Benchmark maintainers should treat engine portability as a reported dimension, not an implementation footnote.