MILM: Large Language Models for Multimodal Irregular Time Series with Informative Sampling

2026-05-13 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Health & Medical Research · Depth: Advanced, medium

Summary

MILM (Multimodal Irregular time series Language Model) is a new approach that leverages Large Language Models (LLMs) to process multimodal irregular time series (MITS) data, such as electronic health records (EHR). MITS data includes asynchronous, irregularly sampled observations from both numerical and textual channels, where timing and channel patterns carry significant predictive signals. MILM represents MITS as time-ordered triplets in Extensible Markup Language (XML) format and fine-tunes an LLM using a two-stage strategy for MITS classification. The first stage trains on value-redacted MITS to learn from sampling patterns, while the second stage models both sampling patterns and observed values. The MILM-2S model and its single-stage counterpart, MILM-Direct, achieved the best and second-best average performance on multiple EHR datasets, with MILM-2S showing a larger performance margin in scenarios where some values are unavailable at prediction time.

Key takeaway

For AI Scientists and Machine Learning Engineers working with complex, asynchronous healthcare data like EHRs, MILM offers a robust method to extract predictive signals. You should consider adopting an XML-based representation for multimodal irregular time series and explore two-stage LLM fine-tuning, especially when dealing with incomplete data, as this approach significantly enhances prediction accuracy by leveraging both data values and their irregular sampling patterns.

Key insights

LLMs can effectively process multimodal irregular time series by encoding sampling patterns and values in XML.

Principles

• Irregular sampling patterns are predictive.
• Two-stage training improves MITS classification.
• XML format enables LLM processing of MITS.

Method

MILM represents MITS as time-ordered XML triplets and fine-tunes an LLM in two stages: first on value-redacted data for sampling patterns, then on full data for joint pattern and value modeling.

In practice

• Encode MITS data into XML for LLM input.
• Implement two-stage fine-tuning for MITS tasks.
• Prioritize sampling pattern learning in initial stages.

Topics

• Multimodal Irregular Time Series
• Large Language Models
• Electronic Health Records
• Informative Sampling Patterns
• Two-Stage Fine-tuning

Code references

• Ladbaby/PyOmniTS

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Towards Verifiable Agentic Data Science: Solving Irregular TSQA Via Tool-Grounded Reasoning — IRTS-ToolBench bridges the gap in evaluating LLMs on real-world irregular time series data by offering a standardized benchmark.
• From Hazard Functions to Language Space: Cox-Supervised Distillation of Survival Risk into a Large Language Model — Large language models can internalize and predict survival risk from structured clinical data via text-based distillation.
• AI-assisted Protocol Information Extraction For Improved Accuracy and Efficiency in Clinical Trial Workflows — AI-assisted RAG significantly boosts clinical trial protocol data extraction accuracy and efficiency over standalone LLMs.
• Large Language Models in Transportation Systems Management and Operations: From Text Reasoning to Multi-modal Decision Support — LLMs and MM-LLMs offer a new mechanism for integrating diverse transportation data into operator-facing decision support systems.
• Label-Aware Pseudo-Training Sample Generation for Text Classification — A new data augmentation method uses LLMs to subtly alter sentences, improving NLP performance with limited data.
• Understanding Key Features of Time Series Foundation Models from Epidemic Forecasting — Heterogeneous pretrained representations improve epidemic forecasting, especially with aligned pretraining domains.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.