DNT at #SMM4H–HeaRD 2026: Leveraging BERT-based Encoders and LLMs for Medical Information Extraction

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing, Health & Medical Research · Depth: Expert, short

Summary

Doan Nhat Tien and Thìn Đặng Văn presented their systems for two tasks at the #SMM4H-HeaRD 2026 workshop. For Task 1, multilingual Adverse Drug Event detection, they fine-tuned BERT-based models (InfoXLM, XLM-RoBERTa) and Qwen3.5-9B using ensemble methods. This achieved 0.8584 macro F1 on the development set and 0.5304 F1 on unseen Farsi. For Task 7, span detection of ClinicalImpacts and SocialImpacts in opioid narratives, a DeBERTa-Large model with simplified labeling performed best. It reached 0.583 relaxed F1 and 0.500 strict F1. Their analysis shows LLMs excel on known languages for Task 1. Conversely, transformer-based models with simplified labeling generalize better for Named Entity Recognition (NER) tasks.

Key takeaway

For NLP Engineers developing medical information extraction systems, consider your language requirements. If you target known languages for Adverse Drug Event detection, fine-tuning LLMs like Qwen3.5-9B can yield high performance. For Named Entity Recognition tasks needing broader generalization, especially across languages or for span detection, prioritize transformer-based models. Using DeBERTa-Large with simplified labeling offers superior adaptability beyond specific language training.

Key insights

LLMs perform well on known languages, while transformer models with simplified labeling generalize better for NER.

Principles

• LLMs excel on known languages.
• Simplified labeling improves NER generalization.
• Ensemble methods boost F1 scores.

Method

Fine-tune BERT-based multilingual models and LLMs with ensemble methods for ADE detection. Apply DeBERTa-Large with simplified labeling for span detection.

In practice

• Use Qwen3.5-9B for known language ADE.
• Apply DeBERTa-Large for span detection.
• Simplify labels for better NER generalization.

Topics

• Medical Information Extraction
• Adverse Drug Event Detection
• Named Entity Recognition
• Large Language Models
• BERT-based Encoders
• Simplified Labeling

Best for: AI Scientist, Research Scientist, NLP Engineer

Related on AIssential

• From Hazard Functions to Language Space: Cox-Supervised Distillation of Survival Risk into a Large Language Model — Large language models can internalize and predict survival risk from structured clinical data via text-based distillation.
• Analysing Lightweight Large Language Models for Biomedical Named Entity Recognition on Diverse Ouput Formats — Lightweight LLMs offer competitive biomedical NER performance, with specific output formats proving more effective than multi-format tuning.
• mdok-style at SemEval-2026 Task 9: Finetuning LLMs for Multilingual Polarization Detection — Finetuning mid-sized LLMs with QLoRA and augmented data improves multilingual online polarization detection.
• OpenMedQ: Broad Open Pretraining for Medical Vision-Language Models — Broad, open medical pretraining data enables state-of-the-art VLM performance with fewer parameters.
• Noise reduction in BERT NER models for clinical entity extraction — A post-processing Noise Removal model significantly boosts clinical NER precision by analyzing probability distributions and contextual features.
• Extracting Problem and Method Sentence from Scientific Papers: A Context-enhanced Transformer Using Formulaic Expression Desensitization — Formulaic expression desensitization and context-enhanced transformers improve problem/method sentence extraction from scientific papers, outperforming baselines.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.