A self-improving SQL agent needs results, not just a holdout harness

A new MIT-licensed project called Self-Improving SQL Agent arrived with a better evaluation shape than many young text-to-SQL repositories—and an important blank where the evidence should be.

The project’s architecture is straightforward: a LangGraph workflow links a question to a bundled music-store schema, generates SQL, applies a SELECT/WITH-only guardrail, executes the query, and retries after executor or guardrail errors. It stores lessons from failures in a persistent SQLite “playbook”; optional Mem0 support handles user preferences, while the same execution path is exposed through MCP. The repository also includes a Streamlit interface, SQLite and PostgreSQL setup, and pytest coverage for basic query rejection and execution behavior.

The evaluation design is the interesting part

The included evaluator defines three modes: one-shot generation without memory; retry without a playbook; and retry plus a playbook trained on 30 learning questions. It then scores all three modes on a separate frozen set of 20 holdout questions. Correctness is execution-based: predicted and gold SQL must return the same normalized rows.

That separation matters. A system that writes lessons after failures can look “self-improving” simply because it later sees the same questions. A learn/holdout split at least creates a way to test whether the stored lessons transfer to unseen questions rather than memorize the training set.

But the repository’s committed evals/results.md contains em dashes for every A/B/C score. It tells users to configure an LLM and run the evaluation themselves. The harness is real; a measured improvement is not yet published.

What buyers and builders should ask next

The first missing artifact is a completed scorecard with model, provider, prompt version, temperature, database backend and run date. Because the holdout contains only 20 questions, one additional correct answer moves the headline accuracy by five percentage points; repeated runs would show whether any gain survives model variance.

The second question is whether the playbook helps for the right reason. Its retriever tokenizes the incoming question, scores stored bullets by word overlap, returns up to eight, and falls back to the first stored bullets if nothing overlaps. An ablation should therefore report not only mode C versus mode B, but also retrieved lessons per question and examples where memory hurts.

This is still a useful new directory entry: the code, license, tests and evaluation scaffold are inspectable. The honest verdict, however, is narrower than the project name. It supplies a reproducible experiment for self-improvement; it does not yet supply evidence that self-improvement occurred.