SMMTech at #SMM4H-HeaRD 2026: Detection of Insomnia in Clinical Notes

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Clinical Natural Language Processing · Depth: Expert, quick

Summary

Team SMMTech participated in the SMM4H-HeaRD 2026 Shared Task 2, focusing on detecting insomnia in clinical notes from the MIMIC-III Clinical Database. Their work involved a comparative architectural study between extractive token-classification models, including BERT, ClinicalBERT, BigBird, and Clinical BigBird, and generative Large Language Models. For the official test phase, SMMTech deployed a 4-bit quantized generative hybrid pipeline utilizing Llama3-Med42-8B to assess its multi-hop reasoning capabilities. While this generative pipeline achieved an F1-score of 0.4783 on Subtask 1 (Classification), it encountered difficulties with exact span matching on Subtask 2. The study highlights the mechanical limitations inherent in zero-shot JSON extraction and emphasizes the critical need to decouple clinical reasoning from character-level span extraction in such tasks.

Key takeaway

For Machine Learning Engineers developing clinical NLP solutions, especially for tasks like insomnia detection in clinical notes, recognize that while generative LLMs like Llama3-Med42-8B can perform well on classification (F1-score 0.4783), they currently struggle with exact span matching. You should consider decoupling clinical reasoning from character-level span extraction, potentially by employing specialized extractive models for precise span identification and generative models for higher-level reasoning, to achieve robust performance.

Key insights

Decoupling clinical reasoning from precise character-level span extraction is crucial for LLM-based clinical NLP.

Principles

• Generative LLMs struggle with exact span matching.
• Decouple reasoning from character-level extraction.
• Zero-shot JSON extraction has mechanical limits.

Method

A 4-bit quantized generative hybrid pipeline using Llama3-Med42-8B was deployed for multi-hop reasoning, compared against encoder-only transformers like BERT and BigBird for clinical span extraction.

In practice

• Use encoder-only models for precise span extraction.
• Quantize LLMs (e.g., 4-bit) for deployment.
• Evaluate LLMs for multi-hop reasoning.

Topics

• Clinical NLP
• Insomnia Detection
• Large Language Models
• Span Extraction
• MIMIC-III
• Model Quantization

Best for: NLP Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Measuring the sensitivity of LLM-based structured extraction to prompt, model, and schema choices in clinical discharge summaries — LLM-based clinical data extraction sensitivity varies significantly with model and schema, especially regarding "absence-versus-silence" distinctions.
• EDEN: A Large-Scale Corpus of Clinical Notes for Italian — EDEN provides the largest Italian clinical note corpus and a CRF-filling benchmark for LLM-based information extraction.
• TalkTag: Fine-Grained Morphosyntactic Error Annotation for Transcribed Speech — TalkTag automates fine-grained morphosyntactic error annotation in spoken transcripts using an LLM, even with scarce data.
• Identifying Signs and Symptoms of Urinary Tract Infection from Emergency Department Clinical Notes Using Large Language Models — Prodigy is an efficient, scriptable annotation tool enabling data scientists to accelerate NLP development.
• Detecting HIV-Related Stigma in Clinical Narratives Using Large Language Models — LLMs can effectively detect HIV-related stigma in clinical notes, with encoder models outperforming generative models in zero-shot settings.
• Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks — English-centric LLMs exhibit significant performance gaps and reliability issues in Arabic medical tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.