Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction

2026-03-25 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Health & Medical Research · Depth: Advanced, quick

Summary

RAVEN (Recurrence-Aware next-Visit EveNt prediction) is a new generative pretraining strategy designed for sequential electronic health record (EHR) data. This model, trained on over one million unique individuals, autoregressively generates tokenized clinical events for a patient's next visit, conditioned on their historical data. A key feature is its regularization for predicting repeated events, addressing a common issue in EHR foundation model evaluations where repeated event tokens can inflate performance metrics. The research also explores scaling behaviors in data-constrained, compute-saturated environments, indicating that increasing model size alone is ineffective without proportional data increases. RAVEN demonstrates zero-shot prediction capabilities for disease incidence, performing comparably to fine-tuned Transformer models and surpassing simulation-based next-token methods, and generalizes to external patient cohorts despite clinical code mapping and feature coverage discrepancies.

Key takeaway

For research scientists developing predictive models with electronic health records, you should consider integrating recurrence-aware regularization into your generative pretraining strategies. This approach, exemplified by RAVEN, can improve the accuracy of next-visit event predictions and prevent inflated performance metrics by distinguishing new disease onsets from recurring events, especially when evaluating zero-shot prediction capabilities.

Key insights

RAVEN is a generative pretraining strategy for EHRs that predicts next-visit events while accounting for recurrence.

Principles

• Regularize repeated event prediction.
• Distinguish new onsets from subsequent occurrences.
• Data volume is critical for model scaling.

Method

RAVEN autoregressively generates tokenized clinical events for a patient's next visit, conditioned on history, with regularization for repeated events to prevent inflated performance metrics.

In practice

• Use RAVEN for zero-shot disease incidence forecasting.
• Apply recurrence regularization in EHR models.
• Prioritize data growth over model size.

Topics

• Generative Pretraining
• Electronic Health Records
• Clinical Event Prediction
• Foundation Models
• Zero-shot Learning

Best for: Research Scientist, AI Researcher, AI Scientist, Machine Learning Engineer

Related on AIssential

• Scaling Generative Foundation Models for Chest Radiography with Rectified Flow Transformers — A 1.3B-parameter generative foundation model for chest radiographs achieves expert-indistinguishable synthesis and controllable editing.
• Predicting Post-Traumatic Epilepsy from Clinical Records using Large Language Model Embeddings — LLM embeddings from clinical records enhance early prediction of Post-Traumatic Epilepsy when fused with tabular data.
• Beyond Classification: A Cough Regression Benchmark for Respiratory Acoustic Foundation Models — Respiratory acoustic foundation models can predict continuous health quantities from cough audio using regression, moving beyond classification.
• Forecasting Medium-Horizon Alzheimer's Disease Progression: Residual Gap-Aware Transformers for 24-Month CDR-SB Change from ADNI Clinical and Biomarker Histories — Combining statistical anchoring with gap-aware residual learning improves medium-horizon Alzheimer's disease progression prediction.
• TreeText-CTS: Compact, Source-Traceable Tree-Path Evidence for Irregular Clinical Time-Series Prediction — TreeText-CTS bridges numerical and language models to provide compact, source-traceable evidence for clinical time-series predictions.
• Benchmarking Machine Learning Architectures for Antimicrobial Stewardship in Pediatric ICUs — Machine learning performance for pediatric antimicrobial stewardship is driven more by target design and data characteristics than model complexity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.