CascadeFilter’s accuracy lead comes with a 6-in-20 certification warning

A new cost-aware semantic-filtering system reports strong benchmark accuracy—and then documents why its statistical guarantee is not ready to be treated as empirically validated.

What the benchmark measured

CascadeFilter routes each predicate–document pair through up to three stages: Qwen3-4B on extracted evidence, the same model on the full document, and GPT-5.5 as an oracle fallback. The paper evaluates six datasets in 13 settings, using a default oracle-relative accuracy target of 0.9 and failure budget of 0.05. Against benchmark reference labels, the full system averages 0.938 accuracy and 0.887 macro-F1, with the highest reported value for both metrics in 12 of 13 settings.

The routing result matters as much as the aggregate score. The mean Stage-3 fallback rate is 14.6%, ranging from 2.5% on CB-Q1 to 31.2% on RV-Q3. On the three settings the authors classify as skewed, accuracy and macro-F1 separate sharply: AD-Q2 reports 0.907 accuracy but 0.594 macro-F1. The certificate controls overall disagreement with the oracle; it does not guarantee macro-F1, precision, recall or minority-class performance.

The paper catches its own contradiction

The authors repeated calibration and deployment over 20 draws for each setting. Their current routing table records nine target violations in 260 draws, or 3.46% pooled. But six violations occur on AD-Q2 alone, producing a 6/20 per-setting violation rate; AD-Q3 contributes another 3/20. The paper explicitly says pooled results cannot establish the per-setting guarantee and asks for per-draw logs to reconcile the implementation.

That is not a small editorial footnote. The theorem promises that the selected policy meets the oracle-relative target with probability at least 1−δ under stated assumptions. The paper says its present experiments do not establish empirical validity of that certificate because the violation counts and run provenance remain unresolved.

Cost is modeled, not audited

CascadeFilter reports a mean nominal cost of $2.968×10⁻⁴ per predicate–record. Removing the evidence stage raises the displayed cost to $5.570×10⁻⁴; removing the full-document stage raises it to $4.871×10⁻⁴. But the authors also say the totals require reconciliation to ensure cached Stage-2 predictions and reused calibration labels are not charged twice. The assigned $0.20 per million tokens for the self-hosted small model is a normalization, not measured GPU operating cost.

The defensible conclusion is narrower than “guaranteed cheap filtering.” CascadeFilter reports a promising operating point and unusually candid failure analysis. Until its per-draw logs and accounting ledger are reconciled, the 0.938 benchmark accuracy is a measured result; the certificate and end-to-end savings remain claims awaiting audit.