Why Do Self-Harm Prediction Models Struggle to Generalise? – Lexical and Semantic Variations in Emergency Department Triage Notes

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Health & Wellbeing — Clinical Care & Medical Practice, Mental Health & Psychological Support, Medical Devices & Health Technology · Depth: Expert, quick

Summary

NLP models designed to detect self-harm from emergency department (ED) triage notes demonstrate strong performance within individual hospitals but often experience significant performance decline when applied across different institutions. A study comparing ED triage notes from two hospitals investigated this generalization challenge by analyzing lexical characteristics, highly associated predictive features, and salient topics. The findings indicate substantial variation in lexical expression and feature importance related to self-harm across the hospitals. Despite consistent core themes like self-poisoning and self-injury, these documentation differences are directly linked to reduced cross-site performance. This research offers crucial insights into how institutional variations impact the accurate identification of self-harm in clinical text and suggests avenues for enhancing model generalizability.

Key takeaway

For NLP engineers deploying self-harm prediction models across multiple healthcare institutions, you must account for significant lexical and feature importance variations in clinical documentation. Your models, even with consistent core themes, will likely struggle with cross-site generalization due to these differences. Prioritize developing adaptive strategies or site-specific fine-tuning to maintain performance and ensure reliable risk assessment in diverse emergency department settings.

Key insights

Lexical and feature variations in ED triage notes hinder self-harm prediction model generalization across hospitals.

Principles

• Institutional documentation differences affect NLP model performance.
• Consistent core themes can have varied lexical expressions.
• Feature importance varies across clinical sites.

Method

Compare ED triage notes across institutions by analyzing lexical characteristics, predictive features, and salient topics to identify generalization barriers.

In practice

• Analyze site-specific lexical variations.
• Identify institution-specific predictive features.
• Develop adaptive NLP models for clinical text.

Topics

• Self-Harm Prediction
• Clinical NLP
• Model Generalization
• Emergency Department Triage
• Lexical Variation
• Feature Importance

Best for: AI Scientist, NLP Engineer, Machine Learning Engineer, Research Scientist

Related on AIssential

• Before the Labels: How Dataset Construction Shapes Suicidality Detection in Clinical Text — Dataset construction choices operationalize suicidality in EHRs, impacting NLP model reliability.
• Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks — English-centric LLMs exhibit significant performance gaps and reliability issues in Arabic medical tasks.
• Predicting relations between SOAP note sections: The value of incorporating a clinical information model — Fine-tuning a RoBERTa-base model on manually annotated clinical data improves NER for SOAP note sections.
• Measuring the sensitivity of LLM-based structured extraction to prompt, model, and schema choices in clinical discharge summaries — LLM-based clinical data extraction sensitivity varies significantly with model and schema, especially regarding "absence-versus-silence" distinctions.
• Anatomy of Data Repositories for the Analysis and Detection of Toxicity in Portuguese — Lexical diversity and annotation practices significantly impact cross-domain generalization in hate speech detection datasets.
• Towards a Universal Dependencies Corpus for Portuguese Epidemiological Reports — Document extraction and representation significantly impact parsing epidemiological texts more than parser selection.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.