ClickHouse’s new benchmark tests analytics agents on production-shaped warehouse work

ClickHouse has released an open-source harness for evaluating analytics agents on production-shaped warehouse work, alongside results from 28 models answering 201 questions derived from traffic to its internal analytics assistant. The company’s Agentic Analytics Benchmark measures answer correctness, estimated API cost and wall-clock time rather than judging a single generated SQL statement.

ClickHouse reports that Claude Fable 5.1 led its run with a 76.6% pass rate. It also says the complete 201-question suite cost about $1 to run with DeepSeek V4 Flash, versus $52 with Fable 5.1, with an 11-percentage-point reduction in correctness. Those figures are vendor-reported results on ClickHouse’s reconstructed warehouse, not a universal model ranking.

A benchmark built from live analytics traffic

The question set began with requests made to DWAINE, ClickHouse’s internal analytics agent. According to the company, curation, schema-compatibility checks, model agreement and legal review reduced 501 candidates to 201. The released methodology reconstructs the warehouse with synthetic but internally consistent data, preserving its schema shape while removing real names and figures. ClickHouse’s test instance contains 18 tables and 865 columns across denormalized marts and a dimensional CRM layer.

Ground truth is produced by having models from three providers solve each question independently; a result is retained when at least two agree. Candidate answers are then scored by a three-provider jury that excludes the candidate’s own model family. ClickHouse acknowledges the weakness in that design: agreement is not human verification, and just 51.7% of its ground-truth answers were unanimous.

The company also tested for benchmark contamination. It reports zero successful entity recoveries in 650 attempts, a fleet-average completion similarity of 0.064, and only one of 1,200 completions reaching 0.5 similarity. Because the full private question set and answers are not published, those contamination results cannot be independently reproduced from the repository alone.

What organizations can actually run

The Apache-2.0 data-agent-mnist repository contains the harness rather than ClickHouse’s private benchmark data or leaderboard. It includes two worked synthetic warehouses and scripts for multi-model annotation, evaluation and statistical analysis. Organizations can supply their own question set, warehouse snapshot, schema prompt and model registry.

That distinction matters for practitioners. Conventional text-to-SQL suites usually provide a schema, expect one query and compare execution against a gold answer. The ClickHouse harness instead lets an agent discover the schema, make multiple queries and be judged on the resulting answer. It therefore targets the operational question teams face when selecting an analytics agent: which model works best on their warehouse, joins and user questions at an acceptable cost.

The repository also names material limitations. It currently runs against local chDB snapshots rather than a live cluster, numeric comparison uses a money-oriented tolerance that may be unsuitable for probabilities or small values, and model judges can disagree between runs. Teams adopting the harness should treat those defaults as implementation choices to audit, not as settled evaluation standards.