Ask Your Data makes its text-to-SQL safety and accuracy claims inspectable

A new repository called Ask Your Data packages a ClickHouse text-to-SQL agent, a Streamlit chat interface and a command-line client with something early projects often omit: committed evaluation cases, an explicit grader and model ablations.

The implementation gives Claude one run_sql tool and asks it to return a structured answer containing an answerability flag, narrative answer, value, unit and SQL. Its context combines columns discovered from ClickHouse with hand-written business notes for metrics and known data quirks. The database account is restricted to SELECT on nine business tables, with a 15-second execution limit and 1,000-row cap; the application adds a second check that accepts only one SELECT or WITH … SELECT statement. A prompt-level policy adds a third layer, while request logs retain SQL, token usage, estimated cost and latency.

A test suite that can fail for more than wrong SQL

The repository’s evaluation note defines 54 cases: 41 answerable questions graded within 1% of SQL-computed truth, eight unanswerable questions that must be declined and five adversarial prompts that must be refused without attempting a write or reading system tables. The grader is deterministic rather than model-judged, and any non-SELECT attempt fails the case.

The author reports 54/54 passes for Claude Opus 5.5 with business notes, at $0.011 per question and 6.3 seconds median latency. Removing the notes reportedly preserved accuracy but raised queries per question from 1.2 to 1.9, cost from $0.011 to $0.017 and median latency from 6.3 to 8.9 seconds. A Haiku 5.5 comparison passed 52/54 at a reported $0.0004 per question and 5.5 seconds.

Those numbers are useful less as a leaderboard result than as an example of what an inspectable small-system evaluation can expose. The write-up says the process caught a Decimal-division error in its own ground truth, a request that hung until timeout, and a saturated initial suite that needed harder cases.

The caveats are part of the entry

The evidence is still narrow. All underlying data is synthetic, the author created the test set, each case was run once and the results have not been independently reproduced. The repository also declares no license in its root or GitHub metadata, so public visibility should not be mistaken for permission to reuse the code.

Ask Your Data therefore enters the directory as a research-stage tool, not a production recommendation. Its stronger contribution is methodological: evaluate answerable, unanswerable and adversarial behavior together; compute truth directly from the database; publish ablations; and test the evaluator itself.