Atlas 0.2.26 catches a 22% sample-size shortfall before trusting its own eval

Atlas 0.2.26 is a small release with a useful warning for anyone building evaluation gates: naming a confidence statistic is not enough if the sample-size arithmetic comes from a different approximation.

The release adds two pieces of evaluation infrastructure for Atlas’s “brain” pipeline: a held-out manifest cutter and a cascade measurement harness. The most important result is not a new model score. It is a correction to the test design before the next stage is allowed to claim one.

Sixty clean examples were not enough

The original acceptance criteria said that zero observed misses in 60 examples would clear a 95% lower bound, while allowing one miss would require about 100 examples. Atlas’s measurement-harness pull request recomputed those thresholds with the Wilson bound that the gate actually uses.

For a perfect result, the pull request says the minimum is 73 examples, not 60. That is roughly a 22% increase. With one miss, the minimum becomes 110 rather than about 100. At the proposed sizes, the Wilson lower bounds were 0.9398 for 60 of 60 and 0.9455 for 99 of 100—both below the required 0.95.

The implementation derives the minima in tests instead of pinning those numbers as unexplained constants. That matters because an evaluation gate can look statistically rigorous while borrowing its sample size from a neighboring approximation.

A smoke score is not a release gate

Atlas also records a stage-0 result before its stage-1 adapter exists. On the repository’s smoke fixture, stage 0 drops 38.10% of episodes at observed recall 1.0000. But there are only eight positive examples, so the reported 95% Wilson lower bound is 0.6756.

The harness labels fixtures as either smoke or evaluation, requires provenance for evaluation sets, and refuses to turn a smoke fixture into a gating verdict. It also reports the composed stage-1 measurement as absent rather than treating a missing adapter as a no-op result. Those distinctions keep “the code ran” separate from “the evidence supports rollout.”

Reproducibility without copying customer content

The held-out-manifest pull request stores episode IDs, classes, cut date, window and dial evidence—not the episode bodies themselves. The project says bundles can contain unredacted Slack messages or email bodies; committing those would create a durable copy outside the platform’s purge mechanisms.

Keeping only identifiers preserves a reproducible set while making deletion visible: if a source row has been purged, verification fails instead of silently evaluating against a stale copied body. The verifier also checks region containment before resolving IDs, records still-draining episodes rather than pretending every row has settled, and labels its dial evidence as region-local rather than fleet-wide.

The practical lesson is broader than Atlas: a credible held-out evaluation needs three boundaries at once—correct confidence math, a hard distinction between smoke and gating datasets, and a reproducibility format that does not become a shadow archive of customer data. Atlas 0.2.26 has built the instrument and corrected its arithmetic. It has not yet produced the stage-1 score, and the release does not claim otherwise.