ClickHouse’s 1,996× benchmark claim starts before the query runs

ClickHouse has published the first end-to-end results from CostBench, a vendor-run cloud warehouse benchmark that measures ingestion, maintenance of query-ready data and scheduled query serving at the same time. ClickHouse says its Cloud service delivered 412–1,996× better performance per dollar than the tested Snowflake, BigQuery and Redshift Serverless configurations. That is an eye-catching result—but the benchmark’s real contribution is the boundary it draws around the system, not the headline multiplier.

What CostBench measures

The published methodology replays a 113.2-billion-row, 12-column stock-quotes dataset toward one million rows per second. While ingestion is active, four aggregate queries run every 10 minutes and two event-level drill-down queries run hourly. Each platform must also maintain symbol-oriented layouts and daily pre-aggregations as new rows arrive.

That design targets a failure mode familiar to analytics-agent builders: rows can become visible before derived data, clustering or rollups have caught up. An agent may therefore issue valid SQL against fresh base data while dashboards or prepared summaries still lag. CostBench makes that preparation work part of the measured system rather than treating query-ready tables as free input.

Its lower-is-better score is (fresh-data-path cost + normalized query-serving cost) × accumulated query runtime. ClickHouse reports that the rival systems were 32–101× worse in its earlier query-only comparison, widening to 412–1,996× once continuous preparation was included.

Read the multiplier carefully

The repository documents an important caveat: Snowflake, BigQuery and Redshift each use a ClickHouse-matched active-ingestion window, and the global relative scores reuse those pairwise comparisons rather than performing one new four-way match. Its absolute cost-versus-runtime chart intentionally combines those distinct windows.

The score also normalizes query cost as accumulated runtime multiplied by a read-side compute rate. ClickHouse says this removes idle timeouts and minimum billing windows, but it is a modeled comparison—not a reconstruction of customer invoices. Storage, discounts, free tiers, idle capacity, minimum billing and failed attempts are excluded; required ingestion infrastructure remains included.

Reproduction is constrained too. The repository publishes scripts, accepted outputs, cost summaries and chart provenance, but not the market-data capture because the supplier’s terms prohibit redistribution. A reproducer must bring a licensed dataset with the same schema.

The practitioner takeaway

Do not transfer the 1,996× figure directly into a procurement spreadsheet. Do transfer the workload model. A warehouse evaluation for analytics agents should run ingest, layout maintenance, pre-aggregation and realistic query traffic concurrently, then measure both answer freshness and the cost of the whole path. CostBench gives teams an auditable starting harness for that test—and enough disclosed boundaries to know which result they still need to verify themselves.