DBcover shows where LLM-guided SQL testing helps—and where coverage stops

DBcover’s headline result is easy to misread. In 24-hour, time-bounded experiments, the framework reached 80.1% line coverage on PostgreSQL 17.0 and 82.3% on MySQL 8.0.33. Those are not text-to-SQL accuracy scores, and they are not proof that either database is 80% correct. They measure how much source code the generated SQL tests executed. DBcover paper

The useful comparison is against the same databases’ existing regression-test seeds. Those seeds covered 68.6% of PostgreSQL and 71.2% of MySQL. DBcover added 11.5 and 11.1 percentage points, respectively. By contrast, the syntax-preserving SQUIRREL fuzzer added only 0.4 points on PostgreSQL and 0.2 points on MySQL in the paper’s setup. ShQveL, an LLM-assisted generator run without the shared seed corpus, reached 31.5% and 25.7%. DBcover paper

The difference is not simply “use an LLM.” DBcover first records which execution paths are reached by existing SQL tests, builds a call graph and links the two in a knowledge graph. For an uncovered target, it selects a seed whose path is nearby, then gives a Qwen3-32B model both that global path context and the target function’s local source context. Removing the global context reduced 24-hour coverage to 75.7% on PostgreSQL and 77.2% on MySQL—drops of 4.4 and 5.1 points. DBcover paper

That ablation is the most transferable result. An LLM asked to invent a test from a function body can produce plausible SQL that never reaches the function. A real seed provides the valid schema, state and call-path neighborhood; the model’s job becomes a constrained mutation rather than blind generation. In the paper’s PostgreSQL example, reaching a partition-cache branch required 15 extra inserts into the same range partition, not a novel query shape. DBcover paper

The plateau matters too. The authors say much of the remaining uncovered code handles crash recovery, I/O failures and server lifecycle events that valid SQL cannot trigger. Other branches require numeric constraint solving beyond the evaluated model. Coverage therefore levels near 80% partly because the input channel is SQL, not because the generator exhausted database behavior. DBcover paper

Practitioners should treat DBcover as evidence for a testing architecture: combine runtime traces, source context and executable feedback before asking a model to generate tests. But attach a bug oracle—differential checks, metamorphic relations or crash monitoring—before calling the result a reliability gain. The paper measures executed lines; systematic bug discovery remains future work. DBcover paper