OmniTable’s 35 PB lesson: LLM data curation is a systems benchmark, too

Ant Group researchers have put numbers on a part of model development that benchmark tables usually omit: the time spent assembling and revising training data. In a PVLDB paper released on arXiv, the team says its OmniTable system manages more than 35 PB across web, code, PDF and supervised fine-tuning data.

What the 5.6× result measures

The paper evaluates an SFT curation job that collects data from eight sources, computes 12 features—nine CPU UDFs and three GPU-inference features—and exports a filtered subset. The authors hold input batches and filtering thresholds fixed, then compare OmniTable with the production workflow it replaced.

The legacy process took about 14 days: roughly 2 days for ingestion, 9.5 days for feature backfill and 2.5 days for export. OmniTable took about 2.5 days: 0.5 days, 1.7 days and 0.3 days for the same stages. That is the paper’s reported 5.6× end-to-end speedup. Manual steps fell from 45 to 12, a 73.3% reduction, while pipelines or commands fell from 24 to 10.

This is not a conventional engine shootout. The legacy baseline is reconstructed from historical task records and operation logs, and much of the gain comes from removing coordination and recovery work: dependency resolution, CPU/GPU routing, operator fusion, parameter tuning and record-level fault isolation. That makes the comparison operationally useful, but not a clean estimate of how much faster one execution engine is than another.

The supporting experiments

The paper separately tests several mechanisms. Fusing eight features over a roughly 2.5 PB Common Crawl batch reduced consumption from 42,000 to 18,500 CPU-hours and shortened elapsed time from 38 hours to 14 hours. On a 500 GB fastText feature job containing 31,247 anomalous records, record-level failover completed 99.995% of about 600 million records in 6.2 hours; disabling failover caused the job to fail, while the legacy process needed three manual cycles and about 52 hours.

For data exploration, OmniTable maintained 18–23 TB/hour on a fixed filtered-export workload as selected data grew from 1 TB to 25 PB. At 25 PB, the no-governance ablation reached about 5 TB/hour and the legacy workflow about 2 TB/hour. On the production web table—25 PB, more than 300 billion records and over 800 logical columns—the global ID index reduced median point-lookup latency from 184 seconds to 8.3 seconds.

The important benchmark lesson is methodological: at production scale, malformed records, layout decay and human retries can dominate the wall clock. OmniTable’s result should be read as a workflow benchmark with a historical baseline, not a universal 5.6× systems claim. Even with that caveat, it is unusually concrete evidence that data-infrastructure design can compress an LLM iteration cycle by days.