Autoregressive EHR Foundation Models with Multimodal Inputs

2026-07-24 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, AI in Healthcare · Depth: Expert, quick

Summary

A new framework is introduced for autoregressive foundation models that integrate multimodal clinical data, including ECG waveforms, chest X-ray images, and clinical notes, with tokenized electronic health records (EHRs). While existing models often rely solely on structured event codes for zero-shot clinical prediction, this framework employs modality-specific latent compression and gated cross-attention with temporal alignment. Researchers investigated two critical design choices: the method for compressing long per-modality sequences to reduce compute overheads and improve generalization, and the influence of pretrained encoders on downstream performance. Controlled ablations on the MIMIC-IV dataset demonstrated that optimal latent-compression configurations surpassed both uncompressed cross-attention and mean pooling. Stronger pretrained encoders consistently yielded better results. However, the study also found that simply adding auxiliary modalities does not inherently improve ICU mortality prediction, underscoring the necessity for meticulous fusion architecture design and appropriate clinical evaluation.

Key takeaway

For Machine Learning Engineers developing clinical prediction models with EHRs, merely incorporating auxiliary modalities like ECGs or X-rays will not guarantee improved outcomes. You must prioritize careful design of the fusion architecture and evaluate latent compression strategies. Focus on selecting robust pretrained encoders and rigorously testing how multimodal inputs are integrated to achieve meaningful improvements in tasks like ICU mortality prediction.

Key insights

Integrating multimodal clinical data into EHR foundation models requires careful design of latent compression and fusion architectures for effective prediction.

Principles

• Latent compression improves generalization.
• Stronger pretrained encoders boost performance.
• Fusion architecture design is critical.

Method

A framework conditions autoregressive EHR models on auxiliary clinical modalities (ECG, X-ray, notes) using modality-specific latent compression, gated cross-attention, and temporal alignment.

In practice

• Test latent compression configurations.
• Prioritize stronger pretrained encoders.
• Rigorously evaluate fusion architectures.

Topics

• Autoregressive Models
• Electronic Health Records
• Multimodal AI
• Latent Compression
• Cross-Attention
• Clinical Prediction
• MIMIC-IV

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Large Language Models as Unified Multimodal Learners for Clinical Prediction — Fine-tuning LLMs on text-serialized multimodal EHR data simplifies clinical prediction architectures while maintaining or improving accuracy.
• Pretraining EHR Foundation Models with Patient-Aware Sampling — Patient Sampling improves EHR foundation model performance by controlling training signal distribution and mitigating data bias during pretraining.
• A Multimodal and Explainable Machine Learning Approach to Diagnosing Multi-Class Ejection Fraction from Electrocardiograms — Multimodal machine learning can accurately stratify LVEF from ECG and EHR data, enhancing accessibility.
• Learning Sparse Latent Predictive Foundation Model for Multimodal Neuroimaging — Neuro-JEPA is a sparse multimodal neuroimaging foundation model using a latent predictive objective and Mixture-of-Experts.
• I Gave a Model Eyes and a Genome. Here’s What It Learned About Brain Cancer. — Cross-attention fusion of pathology and genomics significantly improves glioblastoma survival prediction over unimodal approaches.
• TMF-RSE: Tri-Modal Fusion with Regional Semantics and Evidential Uncertainty for Lung Severity Scoring — Tri-modal fusion with regional semantics and evidential uncertainty improves lung severity quantification.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.