Ensemble Diversity Optimization for Subjective Supervision

2026-07-09 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

Ensemble Diversity Optimization (EDO) is a novel prediction-space framework designed for subjective Natural Language Processing tasks that exhibit systematic annotator disagreement. EDO jointly optimizes ensemble weights, effective cardinality, and calibration through a unified differentiable objective. It learns ensemble composition and size end-to-end using Gumbel-Softmax relaxation and integrates a signed diversity regularizer. This regularizer, tuned on validation data, steers optimization to either preserve or suppress disagreement, preventing ensemble collapse and managing the utility-calibration trade-off. The framework also incorporates a soft F1 surrogate, class-weighted cross-entropy for class imbalance, and reliability-weighted diversity to regulate intra-ensemble variability. Experiments on four subjective text-classification benchmarks—ArMIS, ConvAbuse, HS-Brexit, and MD-Agreement—demonstrate EDO's substantial improvements in probabilistic calibration, reducing cross-entropy by 40-78% and lowering Brier scores compared to Soft-CE, Soft-MD, Top-5 Voting, and WEL, while maintaining competitive F1 scores and better alignment with annotator distributions.

Key takeaway

For Machine Learning Engineers developing models for subjective NLP tasks, Ensemble Diversity Optimization (EDO) offers a robust solution to represent annotator uncertainty. You should consider implementing EDO to significantly improve probabilistic calibration, as it reduces cross-entropy by 40-78% and lowers Brier scores compared to traditional methods. This approach allows your models to better align with human annotator distributions while maintaining competitive F1 performance, providing a more nuanced and accurate representation of subjective data.

Key insights

EDO uses a prediction-space framework with a signed diversity regularizer to model human subjectivity and improve calibration in subjective NLP tasks.

Principles

• Jointly optimize ensemble weights, cardinality, and calibration.
• Use signed diversity regularization to control disagreement.
• Prevent ensemble collapse via diversity regularization.

Method

EDO learns ensemble composition and size end-to-end via Gumbel-Softmax relaxation. It integrates a signed diversity regularizer, soft F1 surrogate, class-weighted cross-entropy, and reliability-weighted diversity.

In practice

• Apply EDO to subjective text classification.
• Improve probabilistic calibration in NLP models.
• Model annotator disagreement effectively.

Topics

• Ensemble Diversity Optimization
• Subjective NLP
• Probabilistic Calibration
• Text Classification
• Gumbel-Softmax
• Annotator Disagreement

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• emb-diversity: A Tool for Embedding-Based Measurement of Data Diversity — emb-diversity standardizes embedding-based data diversity measurement for fair, robust NLP models across various diversity notions.
• CAMO: A Class-Aware Minority-Optimized Ensemble for Robust Language Model Evaluation on Imbalanced Data — CAMO is a hierarchical ensemble method that dynamically boosts minority class predictions using confidence and uncertainty.
• Reducing Human Annotation with ML Active Learning — Active learning efficiently trains models by selectively labeling the most informative data points, reducing annotation costs.
• Nürnberg NLP at PsyDefDetect: Multi-Axis Voter Ensembles for Psychological Defence Mechanism Classification — Error independence in multi-voter ensembles is key for ambiguous classification tasks with overlapping categories, outperforming single stronger models.
• Quantifying and Predicting Disagreement in Graded Human Ratings — Annotation disagreement in graded human ratings can be predicted from textual features, especially for inappropriate language.
• SMASH at SemEval-2026 Task 9: Detecting Multilingual Polarisation with Encoder Ensembles and Calibrated Decision Thresholds — Language-adaptive ensembles with calibrated decision thresholds significantly improve multilingual polarisation detection performance.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.