Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles

2026-07-25 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

An audit of five diversity measures in Large Language Model (LLM) ensembles reveals that these metrics primarily re-express model capability rather than true diversity. Analyzing 31,900 subsets of 30 LLMs on MMLU-Pro and 29 models on TruthfulQA, researchers found that while latent complementarity exists in 100% of subsets, simple majority voting rarely outperforms the best individual member, achieving this in only 9.98% of size-3 subsets. Critically, "strict diversity" is nearly collinear with one minus mean accuracy (Spearman ρ=+0.991), indicating strong entanglement with capability. Furthermore, three common contingency-table statistics are algebraically non-separable. After capability control, a modest, directionally robust, but configuration-dependent negative association between pairwise co-failure (shared errors) and majority-vote gain persists, suggesting more shared errors lead to lower gain.

Key takeaway

For Machine Learning Engineers designing LLM ensembles, relying solely on raw diversity metrics to predict performance gain is misleading. You should rigorously control for member capability and its spread, including nonlinear specifications, when evaluating ensemble strategies. Recognize that common contingency-table diversity measures are algebraically coupled and that increased shared errors consistently reduce majority-vote gain. Prioritize validating ensemble selection on held-out items to ensure actual performance improvements over the strongest individual model.

Key insights

LLM ensemble diversity metrics are largely confounded by model capability, not reliably predicting majority-vote gain.

Principles

• Latent complementarity exists in 100% of LLM subsets.
• LLM diversity measures are often entangled with model capability.
• Increased shared errors correlate with reduced majority-vote gain.

Method

Audited five diversity measures as predictors of majority-vote gain over the best member, using capability-controlled linear rank-space residualization.

In practice

• Always evaluate ensembles against their strongest member.
• Control for LLM capability before crediting diversity metrics.
• Validate ensemble selection using held-out item sets.

Topics

• LLM Ensembles
• Diversity Metrics
• Majority Voting
• Model Capability
• Error Correlation
• MMLU-Pro Benchmark

Best for: Research Scientist, AI Architect, AI Engineer, AI Scientist, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles — Diversity metrics for LLM ensembles often re-express capability, not true diversity, limiting majority-vote gain.
• Categorizing Mathematical Concepts with LLM Voting Ensembles in Mathswitch — LLM voting ensembles effectively filter noisy mathematical concept data from collaborative knowledge graphs like Wikidata.
• LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning — Cross-model consensus, leveraging error decorrelation among diverse LLMs, effectively verifies reasoning chains, often outperforming trained reward models.
• emb-diversity: A Tool for Embedding-Based Measurement of Data Diversity — Embedding-based diversity measurement offers a flexible, standardized approach to quantify various aspects of data diversity across modalities.
• Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference — PRECISE offers statistically unbiased ranking evaluation by combining limited human labels with extensive LLM judgments.
• What's the theoretical basis for using llm consensus as a probability estimator for real world events [R] — LLM consensus for event probability estimation faces challenges with correlated errors and out-of-distribution events.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.