Pretraining EHR Foundation Models with Patient-Aware Sampling

2026-07-24 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Health & Medical Research · Depth: Expert, quick

Summary

A new pretraining sequence-construction method, Patient Sampling, addresses biases in autoregressive electronic health record (EHR) foundation models. Traditional language modeling approaches concatenate patient trajectories into a single token stream, which can mix multiple patients within training windows and over-represent patients with longer records. Patient Sampling allows for controlled distribution of training signals across patients, mitigating these issues. When compared against the standard Global Stream method, stochastic Patient Sampling with controllable weighting demonstrated improved performance on real-world EHR data. Specifically, it enhanced Macro AUROC and AUPRC across various downstream clinical tasks on both MIMIC-IV v2.2 and v3.1 datasets. These findings highlight training and validation sequence construction as a critical, yet underexplored, design choice for developing more robust EHR foundation models.

Key takeaway

For Machine Learning Engineers developing autoregressive EHR foundation models, you should re-evaluate your pretraining sequence construction. Standard methods risk biasing models towards longer patient records and mixing patient data. Adopting Patient Sampling, which controls training signal distribution, can significantly improve your model's Macro AUROC and AUPRC on clinical tasks, as demonstrated on MIMIC-IV v2.2 and v3.1. Prioritize this design choice to build more robust and equitable EHR models.

Key insights

Patient Sampling improves EHR foundation model performance by controlling training signal distribution and mitigating data bias during pretraining.

Principles

• Longer EHRs bias model optimization.
• Mixed patient data degrades training.
• Sequence construction impacts EHR model performance.

Method

Patient Sampling is a pretraining sequence-construction method for EHR foundation models. It controls training signal distribution across patients using stochastic sampling with controllable weighting, preventing bias from concatenated trajectories.

In practice

• Implement Patient Sampling for EHR models.
• Prioritize sequence construction in EHR pretraining.
• Evaluate Macro AUROC/AUPRC with Patient Sampling.

Topics

• EHR Foundation Models
• Patient Sampling
• Pretraining Techniques
• Clinical Prediction
• MIMIC-IV Dataset
• Bias Mitigation

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Autoregressive EHR Foundation Models with Multimodal Inputs — Integrating multimodal clinical data into EHR foundation models requires careful design of latent compression and fusion architectures for effective prediction.
• Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction — RAVEN is a generative pretraining strategy for EHRs that predicts next-visit events while accounting for recurrence.
• X-FEMR: A Token-level Explainable Approach for Electronic Health Records Foundation Models using Transformer-based Models — X-FEMR provides token-level explainability for black-box Electronic Health Records Foundation Models using a Transformer-based surrogate.
• Evaluation of EEG Foundation Models for Event-Based Burst-Suppression Detection in ICU — EEG Foundation Models show promise for robust, patient-agnostic burst-suppression detection in ICU settings, even with limited data.
• Vision Foundation Models in Radiology: A Scoping Review of Data, Methodology, Evaluation and Clinical Translation — Radiology-specific Vision Foundation Models show promising transferability but face significant hurdles for clinical translation.
• A Patient Simulation Framework for Risk Assessment of Conversational Healthcare AI: Evaluation of an Antidepressant Decision Aid — A patient simulation framework effectively identifies conversational healthcare AI risks by modeling diverse patient profiles.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.