Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles

2026-07-22 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A new audit investigates whether five common diversity measures for Large Language Model (LLM) ensembles truly track diversity or primarily reflect model capability. Analyzing 31,900 subsets of 30 LLMs on MMLU-Pro and 29 on TruthfulQA, the study reveals three key findings. First, latent complementarity is widespread, with oracle gain positive in 100% of subsets, yet simple majority voting surpasses the strongest member in only 9.98% of canonical size-3 subsets (18.71% with held-out best selection), and a pooled size-2-4 rate of 1.27%. Second, a "strict diversity" proxy, based on joint correctness, shows near collinearity with one minus mean accuracy (Spearman rho = +0.991 / +0.988), indicating strong capability entanglement in raw diversity-gain associations. Third, three linear contingency-table statistics are algebraically non-separable; after capability control, a modest, robust residual pairwise co-failure association emerges, where increased shared error correlates with lower gain, though its magnitude varies.

Key takeaway

For Machine Learning Engineers building LLM ensembles, relying solely on current diversity metrics to select models is misleading. Your ensemble's performance gain from majority voting is likely limited because these metrics often reflect model capability rather than true complementary diversity. Focus on explicitly controlling for individual model capabilities and identifying models with genuinely distinct error patterns to maximize the benefits of ensemble methods.

Key insights

Diversity metrics for LLM ensembles often re-express capability, not true diversity, limiting majority-vote gain.

Principles

• Latent complementarity is ubiquitous in LLM ensembles.
• Strict diversity metrics are highly correlated with mean accuracy.
• Shared error among ensemble members reduces gain.

Method

The study audited five diversity measures by predicting majority-vote gain over the best member across 31,900 subsets of 30 LLMs on MMLU-Pro (29 on TruthfulQA) under explicit capability controls.

In practice

• Re-evaluate diversity metrics for LLM ensemble selection.
• Focus on reducing shared errors among models.
• Consider capability controls when assessing ensemble performance.

Topics

• LLM Ensembles
• Diversity Metrics
• Majority Voting
• Model Capability
• MMLU-Pro Benchmark
• Ensemble Performance

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles — LLM ensemble diversity metrics are largely confounded by model capability, not reliably predicting majority-vote gain.
• emb-diversity: A Tool for Embedding-Based Measurement of Data Diversity — emb-diversity standardizes embedding-based data diversity measurement for fair, robust NLP models across various diversity notions.
• Categorizing Mathematical Concepts with LLM Voting Ensembles in Mathswitch — LLM voting ensembles effectively filter noisy mathematical concept data from collaborative knowledge graphs like Wikidata.
• Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference — PRECISE offers statistically unbiased ranking evaluation by combining limited human labels with extensive LLM judgments.
• What's the theoretical basis for using llm consensus as a probability estimator for real world events [R] — LLM consensus for event probability estimation faces challenges with correlated errors and out-of-distribution events.
• UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs — LLMs struggle significantly with generating samples calibrated to true underlying distributions, hindering their use in simulations.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.