A 0.716 MMLU ensemble score hides a rule that can reverse the result

The result

A new agent-aggregation study reports that label-free BP-TD with co-occurrence weighting reached 0.716 accuracy, versus 0.624 for majority vote, on test halves of a 160-question MMLU sample. The protocol used 13 locally served agents built from five base-model families, split each dataset 40 times into validation and test halves, and applied corrected resampled t-tests with Holm adjustment. The gain is therefore more informative than a single full-set score—but it is also unusually dependent on who is in the voting pool.

The pool was deliberately correlated. Its largest family contained six Llama 3.2 agents scoring roughly 0.43–0.49, while four Qwen 2.5 agents formed the strongest family at roughly 0.76–0.83 on MMLU. BP-TD discounts agents that share provenance or answer patterns so a family contributes about one effective vote. In this setup that prevents six weaker variants from outvoting fewer stronger agents.

The boundary matters more than the headline

Across 713 sub-pools per benchmark—2,852 across MMLU, SciQ, TruthfulQA and ARC—the direction reversed when the largest family was also the most accurate. On MMLU, family-weighted BP-TD beat majority vote in 98% of random sub-pools where the largest group was weaker than the rest, with a median gain of 0.10. When the largest group was strongest, it lost in 84% of random and 87% of constructed sub-pools, with median changes of −0.03 and −0.05.

That is not a footnote. It means decorrelation is not intrinsically better aggregation; it is a correction for one specific failure mode: redundant weak agents manufacturing a majority. Discounting a redundant strong family throws away useful votes.

The authors tested two label-free diagnostics for distinguishing those regimes. Neither generalized: pooled AUC was 0.53 for agreement between the largest group and the remaining agents, and 0.51 for the fraction of majority-vote answers changed by BP-TD. The paper therefore does not offer an unsupervised way to know in advance whether its correction is safe.

A few labels beat a clever ensemble

The most practical result comes from the label-budget experiment. Selecting the best single agent using just five labeled MMLU questions overtook label-free family-weighted BP-TD; with 20 labels, the selected agent reached 0.805, versus 0.711 for BP-TD family on the 160-question sample. Tuning the aggregation threshold with very few labels was worse: two, five and ten labels produced 0.600, 0.637 and 0.669, all below the 0.713 label-free Otsu setting.

The larger MMLU harvests make the result less fragile but not universal. On 10,000 questions, BP-TD family scored 0.707 against 0.674 for majority vote, while the one-pass weighted vote scored 0.702 and ran in under one second on CPU. Only MMLU was harvested at that scale; all tested agents were local open models no larger than 14B parameters, and benchmark contamination could alter the family-level correlation structure.

The defensible takeaway is narrow: use dependence-aware weighting when labels are unavailable and there is reason to believe the biggest agent family is not the best one. If even a small labeled set is available, selecting a strong agent may be safer than aggregating them all.