Large Language Models as Unified Multimodal Learners for Clinical Prediction

2026-07-20 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Clinical AI · Depth: Advanced, extended

Summary

A new approach for clinical prediction using Large Language Models (LLMs) involves converting all patient data, including free-text clinical narratives and structured measurements like vital signs and lab values, into a single natural language sequence. This sequence then fine-tunes a pretrained LLM end-to-end, eliminating the need for complex, task-specific multimodal fusion architectures. The method was evaluated across three distinct clinical prediction tasks: in-hospital mortality on MIMIC-III, graft failure prediction from a German transplant center, and emergency triage classification from ambulance records. Using encoder-based ModernBERT and decoder-based models like Llama 3.1, Gemma 2B, DeepSeek-R1-Qwen 8B, and Qwen 3 8B, the unified textual serialization consistently matched or surpassed task-specific multimodal baselines. Notably, it outperformed a clinically deployed gradient boosting system for graft failure prediction, achieving 0.90 AUROC compared to 0.89 AUROC, and improved AUPR from 0.43 to 0.47. Decoder-only models, such as DeepSeek-R1, also demonstrated superior robustness in noisy text environments like emergency triage, achieving a Macro F1 of 0.56 versus the fusion baseline's 0.40. This approach significantly reduces system complexity and engineering overhead.

Key takeaway

For AI Scientists and Machine Learning Engineers developing clinical prediction systems, you should consider adopting a unified text serialization approach for multimodal Electronic Health Record (EHR) data. This method, demonstrated to match or exceed specialized fusion architectures and outperform a deployed gradient boosting model, drastically simplifies system design. Prioritize evaluating decoder-only LLMs for tasks involving noisy or abbreviated clinical text, but be mindful of increased context window demands for extensive patient histories.

Key insights

Fine-tuning LLMs on text-serialized multimodal EHR data simplifies clinical prediction architectures while maintaining or improving accuracy.

Principles

• Unified text serialization eliminates bespoke fusion architectures.
• LLMs can effectively learn cross-modal relationships via self-attention.
• Decoder-only LLMs show robustness with noisy, sparse clinical text.

Method

Structured EHR variables (demographics, labs, vitals) are serialized into key-value textual representations and concatenated with clinical narratives. This unified sequence fine-tunes pretrained LLMs end-to-end for prediction.

In practice

• Convert structured EHR data to key-value text for LLM input.
• Fine-tune decoder-only LLMs for noisy, short clinical notes.
• Evaluate serialization against existing clinical decision support.

Topics

• Large Language Models
• Clinical Prediction
• Electronic Health Records
• Multimodal Learning
• Text Serialization
• Medical AI
• MIMIC-III

Best for: NLP Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Large Language Models as Unified Multimodal Learners for Clinical Prediction — Converting all patient data to text for LLM fine-tuning simplifies multimodal clinical prediction while maintaining performance.
• ChatHealthAI: Aligning Electronic Health Record Representations with Large Language Models for Grounded Clinical Reasoning — ChatHealthAI integrates EHR representations with LLMs for grounded, interpretable clinical reasoning and accurate patient prediction.
• v333: Proceedings of CHIL 2026 — Machine learning, particularly LLMs, is rapidly expanding its utility across diverse clinical and administrative healthcare applications.
• LLM4EHR: Aligning Clinical Time Series with Medical Event Sequences via Large Language Models — Aligning clinical time series with medical event sequences via foundation models significantly improves clinical outcome predictions.
• Language Models as Interfaces, Not Oracles: A Hybrid LLM-ML System for Pediatric Appendicitis — Hybrid AI systems improve clinical decision support by using LLMs for feature extraction and validated ML models for reliable, auditable prediction.
• Predicting Post-Traumatic Epilepsy from Clinical Records using Large Language Model Embeddings — LLM embeddings from clinical records enhance early prediction of Post-Traumatic Epilepsy when fused with tabular data.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.