Diverse Transformer Ensemble with Majority Voting for Medical Decision Extraction at MedExACT 2026

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Natural Language Processing · Depth: Expert, medium

Summary

Rishik Kondadadi's system for the MedEx-ACT 2026 shared task achieved a 0.5554 score on the test set for extracting and classifying medical decisions from ICU discharge summaries. The system frames the task as BIO token classification and employs an ensemble of 25 diverse transformer models, encompassing 13 distinct architectures such as Longformer, DeBERTa, RoBERTa, BioBERT, and SciBERT. Each model is trained using category-aware oversampling, focal loss, and demographic-group-aware sampling to address class imbalance and ensure fairness across patient subgroups. During inference, predictions are aggregated via text-normalized majority voting, where spans are retained if agreed upon by at least 6 of the 25 models. This approach demonstrated that a simple vote-based ensemble with architectural diversity outperforms more complex filtering methods, highlighting architectural diversity as a key driver of ensemble quality and cross-validation as crucial for model selection on small clinical datasets.

Key takeaway

For NLP Engineers building robust medical text extraction systems, consider implementing architecturally diverse transformer ensembles. Your systems can achieve higher accuracy by combining models like Longformer and BioBERT with majority voting, as demonstrated by the 0.5554 score on MedEx-ACT 2026. Ensure you incorporate category-aware oversampling and demographic-group-aware sampling during training to improve fairness and handle class imbalance, and use cross-validation for reliable model selection, especially with small clinical datasets.

Key insights

Architecturally diverse transformer ensembles with majority voting effectively extract medical decisions, outperforming complex filtering.

Principles

• Architectural diversity enhances ensemble quality.
• Cross-validation is vital for small clinical datasets.
• Address class imbalance and fairness in training.

Method

Frame medical decision extraction as BIO token classification. Train 25 diverse transformer models with oversampling, focal loss, and demographic-aware sampling. Aggregate predictions via text-normalized majority voting, requiring 6/25 model agreement.

In practice

• Use diverse transformer architectures in ensembles.
• Implement majority voting for robust predictions.
• Apply demographic-aware sampling for fairness.

Topics

• Transformer Ensembles
• Medical Decision Extraction
• BIO Token Classification
• Majority Voting
• Clinical NLP
• Model Fairness

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• TeamHerald@CHIPSAL 2026: Hate Speech Detection and Sentiment Analysis of Nepali Memes using Transformer-based Architectures and Ensemble Learning — Ensemble strategies for meme analysis perform differently for binary vs. multi-class tasks, requiring task-specific aggregation.
• Severity-Aware Curriculum Learning with Multi-Model Response Selection for Medical Text Generation — Severity-aware curriculum learning with multi-model response selection significantly improves medical text generation quality and relevance.
• State-of-the-Art Transformer Pipelines in spaCy — spaCy offers robust, configurable Transformer pipelines for NLP tasks, often outperforming LLMs on predictive tasks with modest data.
• Automatic Evaluation of Enem Essays: A Comparative Analysis between Feature Engineering and Transformers — Regression formulations are generally more suitable for ordinal essay scoring than multiclass classification.
• EDEN: A Large-Scale Corpus of Clinical Notes for Italian — EDEN provides the largest Italian clinical note corpus and a CRF-filling benchmark for LLM-based information extraction.
• Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks — English-centric LLMs exhibit significant performance gaps and reliability issues in Arabic medical tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.