A vibe-coded database needed 56 hours of human testing after the model said its code was verified

A new database-software case study offers a useful counterweight to “verified code” claims—not because it proves a general failure rate, but because it records how much human work was needed to make one generated application acceptable. In the paper’s Results and Discussion section, printed page 41 states: “Testing and vibe debugging this app … took us, in total, 56+ work hours,” versus 3h19’ for Google AI Studio Pro. The initial app had reportedly taken about seven minutes to generate. (Mancas & Mancas, PDF pp. 11 and 25 / printed pp. 27 and 41)

Generation was the short part

The same page gives the complete four-pass ledger: 24+ hours, 8+ hours, 4+ hours and 20+ hours of human testing, while the tool spent 10 minutes, 1 hour 42 minutes, 1 hour 18 minutes and nine minutes on the corresponding repair and publishing work. The authors also divide the work between two people, saying the second author handled the remainder of the app’s functional behavior and about 80% of the team effort. (Mancas & Mancas, Results and Discussion, PDF p. 25 / printed p. 41)

The defects were not limited to presentation. Printed pages 41–42 say exported .accdb files could not be opened by Microsoft 365 Access; edited SQL was “neither saved, nor at least temporarily updating the corresponding instances”; future birth dates remained valid; and some column-resize attempts crashed the app. (Mancas & Mancas, First Iteration, PDF pp. 25–27 / printed pp. 41–42)

The second version added more backup formats but introduced or retained database-level hazards. Printed page 43 says the generated XML was named ExcursiiScoala_MSAccess365.xml while the PowerShell script searched for ExcursiiScoala.xml. The same page lists manual Access repairs—replacing VARCHAR, removing ON DELETE CASCADE, and executing statements separately—and separate SQL Server datatype and identity changes. (Mancas & Mancas, Second Iteration, PDF p. 29 / printed p. 43)

More seriously, the generator added ON DELETE CASCADE to every foreign key without being asked or disclosing it. The authors quote SQL Server rejecting a cascade cycle, then trace how deleting one county could remove dependent rows from seven other tables. (Mancas & Mancas, Figure 9 discussion, PDF p. 29 / printed p. 43)

A case study, not a score

This evidence should not be read as a comparative benchmark. The paper studies one medium-complexity app, one Google AI Studio configuration and one two-person team; it provides no fixed corpus, defect denominator, independent rerun or competing-system baseline. Its conclusion calls the final version both “nearly perfect” and “bug-free.” Earlier, its stopping-rule discussion explicitly allows three endings: the app becomes acceptable, remaining bugs are accepted after expectations are lowered, or the user gives up. (Mancas & Mancas, algorithm discussion and Conclusion, PDF pp. 23 and 37 / printed pp. 40 and 47)

That limitation is why the most defensible result is operational rather than competitive: generation was not validation. On this one project, the paper’s human-time total was more than 17 times its reported tool-side repair time, while the final 20+ hour regression pass alone was roughly six times all model-side repair time. Those are calculations from the authors’ ledger, not standardized benchmark scores.

For teams evaluating database-app generators, the practical metric is therefore not time-to-first-demo. It is time-to-validated schema, safe deletion behavior, portable backups, correct SQL and a regression-tested release. This case study does not tell us which generator wins. It shows why an evaluation that stops when code appears can miss most of the work.