A SQL agent found three-minute planner stalls in every database it tested

A new database study supplies a missing test for SQL agents: how long the database spends planning generated SQL before execution begins. Its lightweight GPT-5.5 search found at least one query requiring more than three minutes to plan on every one of seven tested DBMSes—four anonymized commercial systems plus DuckDB, MySQL and PostgreSQL. That is not an execution-accuracy score, and the study has no train/test split; it is a planner-robustness stress test on TPC-H scale factor 100. Source

The benchmark changes what “safe SQL” means

The agent received one tool: submit SQL and receive its planning time. Each search ran for 200 LLM turns. Candidates below five seconds were rejected and the agent was asked to try again; qualifying queries were deduplicated first with SQL-structure features and then with an LLM similarity judge. Commercial systems ran with 4 vCPUs, 32 GiB of memory and default configurations, while planning time was measured through EXPLAIN or its equivalent. Source

At the paper’s strictest threshold, each DBMS produced at least one distinct pattern above three minutes. One anonymized analytical system produced 25; another commercial system, DuckDB and PostgreSQL produced one each. The authors explicitly warn that these counts are search outcomes, not a quality ranking: the agent may explore different regions of the SQL space on different systems. Source

The most operationally useful result is that the stalls were usually system-specific. A TPC-H-derived query took about 14 minutes to EXPLAIN on one commercial DBMS but 6 milliseconds on DuckDB. Across the cross-system matrix, many queries that needed minutes on the discovery system planned in under five seconds elsewhere. The recurring families included correlated subqueries, CTE expansion, repeated subquery expressions, disjunctive joins and constant folding. Source

Add a planning-latency gate

NL2SQL evaluations normally stop at correctness or execution success. This study shows why a production acceptance test should also time EXPLAIN under the target engine and version. Report a planning-latency distribution and timeout rate separately from execution latency; preserve the exact generated SQL; and run the same pathology families after database upgrades. A query can be semantically valid yet tie up planner CPU or memory without executing useful work.

The authors released the discovered queries and parameterized pathology families, making that regression gate reproducible today. The repository currently contains separate found-queries and pathologies directories. Source

The headline is therefore not that one planner is worse. It is that agent-generated SQL opens a robustness axis that execution accuracy does not measure—and every tested engine had a failure on it.