SIEMENS at #SMM4H–HeaRD 2026: The Impact of Training Strategy and Backbone Selection on BERT-based Multilingual Clinical NER

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing, AI in Healthcare · Depth: Expert, short

Summary

SIEMENS participated in the MultiClinNER subtask of the MultiClinAI shared task at the #SMM4H-HeaRD Workshop at ACL 2026. This task involved identifying DISEASE, SYMPTOM, and PROCEDURE mentions in clinical case reports across seven languages: Czech, Dutch, English, Italian, Romanian, Spanish, and Swedish. Researchers compared two BERT-based sequence labeling methods: sentence-level token classification with a fixed train/validation split, and paragraph-level chunking with 5-fold cross-validation and checkpoint merging. They utilized both language-specific BERT models and the multilingual XLM-RoBERTa-large as backbones. The study found that the 5-fold training strategy with checkpoint merging consistently outperformed the fixed split, primarily due to improved training-set coverage. Language-specific BERT encoders were most effective for Spanish and English, while XLM-RoBERTa-large achieved the strongest results for the other five languages via cross-lingual transfer.

Key takeaway

For NLP Engineers developing multilingual clinical Named Entity Recognition (NER) systems, you should prioritize training strategies that maximize data exposure. Implement 5-fold cross-validation with checkpoint merging to improve model robustness and performance, as this approach significantly enhances training-set coverage. Strategically select your backbone: use language-specific BERT for high-resource languages like Spanish and English, and consider XLM-RoBERTa-large for effective cross-lingual transfer in other languages.

Key insights

5-fold training with checkpoint merging and strategic backbone selection significantly improves multilingual clinical NER.

Principles

• Increased training-set coverage boosts NER performance.
• Language-specific BERT excels for high-resource languages.
• XLM-RoBERTa-large enables cross-lingual transfer for others.

Method

Compared sentence-level token classification with fixed split against paragraph-level chunking using 5-fold cross-validation and checkpoint merging for BERT-based sequence labeling.

In practice

• Implement 5-fold cross-validation for robust training.
• Merge checkpoints to enhance model generalization.
• Select backbones based on language resource availability.

Topics

• Multilingual NER
• Clinical NLP
• BERT models
• XLM-RoBERTa-large
• Cross-validation
• Training strategies

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• KliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts — Domain-specific pre-training significantly enhances BERT models for Norwegian clinical NLP tasks.
• Zero-Shot to Full-Resource: Cross-lingual Transfer Strategies for Aspect-Based Sentiment Analysis — Cross-lingual transfer strategies for ABSA vary in effectiveness based on model architecture and resource availability.
• The F1 of Formula One: Applicability of Pre-trained NER Models to Brazilian TV Interview Transcripts — LLMs offer superior linguistic sensitivity for NER in noisy interview transcripts compared to statistical methods.
• When Multiple Scripts Matter: Evaluating ASR in Clinical Settings — Multiscript variability in non-English clinical ASR requires specialized evaluation and script unification during training for accurate performance.
• Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks — English-centric LLMs exhibit significant performance gaps and reliability issues in Arabic medical tasks.
• Predicting relations between SOAP note sections: The value of incorporating a clinical information model — Fine-tuning a RoBERTa-base model on manually annotated clinical data improves NER for SOAP note sections.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.