Large Language Models as Unified Multimodal Learners for Clinical Prediction

2026-07-16 · Source: Artificial Intelligence · Field: Health & Wellbeing — Clinical Care & Medical Practice, Medical Devices & Health Technology, Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new clinical prediction approach unifies multimodal patient data by converting all information, including free-text narratives, vital signs, laboratory values, and comorbidities, into a single natural language sequence. This serialized data then fine-tunes a pre-trained Large Language Model (LLM) end-to-end, eliminating the need for complex, task-specific fusion architectures. Researchers evaluated this method across three distinct clinical prediction tasks: in-hospital mortality on MIMIC-III, graft failure prediction using longitudinal data from a German transplant center, and emergency triage classification from ambulance records. The study compared encoder-based (ModernBERT) and decoder-based (Llama 3.1, Gemma, DeepSeek-R1-Qwen, Qwen3) LLMs against established multimodal baselines and a clinically deployed gradient boosting model. Results consistently showed that the unified textual serialization matched or exceeded specialized multimodal baselines and outperformed the gradient boosting system for graft failure prediction, demonstrating reduced system complexity without performance compromise.

Key takeaway

For AI Scientists or Machine Learning Engineers developing clinical prediction systems, this research suggests a significant simplification in architecture. You can achieve competitive performance by serializing all patient data into a single natural language sequence and fine-tuning a pre-trained Large Language Model, rather than building bespoke multimodal fusion models. This approach reduces development complexity and accelerates deployment for critical tasks like in-hospital mortality or graft failure prediction, offering a more streamlined path to robust clinical AI.

Key insights

Converting all patient data to text for LLM fine-tuning simplifies multimodal clinical prediction while maintaining performance.

Principles

• Unified textual serialization simplifies multimodal learning.
• LLMs can process diverse clinical data without architectural changes.
• Performance matches or exceeds task-specific fusion systems.

Method

Convert all patient data (text, vitals, labs) into a single natural language sequence. Fine-tune a pre-trained LLM end-to-end on this sequence for prediction.

In practice

• Apply textual serialization to diverse EHR data.
• Fine-tune Llama 3.1 or Gemma for clinical tasks.
• Replace complex fusion architectures with LLM fine-tuning.

Topics

• Large Language Models
• Clinical Prediction
• Multimodal Learning
• Electronic Health Records
• Textual Serialization
• Graft Failure Prediction

Best for: NLP Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Large Language Models as Unified Multimodal Learners for Clinical Prediction — Fine-tuning LLMs on text-serialized multimodal EHR data simplifies clinical prediction architectures while maintaining or improving accuracy.
• Overview of #SMM4H-HeaRD 2026 – Task 6: Predicting TNM staging from pathology reports — General-purpose LLMs infer clinical staging from context more robustly than fine-tuned encoders, which excel at direct extraction.
• v333: Proceedings of CHIL 2026 — Machine learning, particularly LLMs, is rapidly expanding its utility across diverse clinical and administrative healthcare applications.
• Language Models as Interfaces, Not Oracles: A Hybrid LLM-ML System for Pediatric Appendicitis — Hybrid AI systems improve clinical decision support by using LLMs for feature extraction and validated ML models for reliable, auditable prediction.
• ChatHealthAI: Aligning Electronic Health Record Representations with Large Language Models for Grounded Clinical Reasoning — ChatHealthAI integrates EHR representations with LLMs for grounded, interpretable clinical reasoning and accurate patient prediction.
• Predicting Post-Traumatic Epilepsy from Clinical Records using Large Language Model Embeddings — LLM embeddings from clinical records can predict post-traumatic epilepsy with high accuracy.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.