Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding

2026-05-22 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

Chronicle is a compact 324M-parameter decoder-only transformer designed for joint natural language and time series understanding. Unlike previous multimodal models that adapt pretrained language models, Chronicle is trained from scratch on both modalities within a single unified architecture, sharing core transformer components. Its pretraining primarily uses unimodal batches, followed by a brief alignment stage with interleaved data. Evaluated against dedicated foundation models in both domains, Chronicle matches Gemma-3-270M-PT on 19 NLU tasks and establishes a new benchmark for frozen-embedding time series classification on 24 UCR/UEA datasets, achieving 0.736 accuracy and 0.712 F1. Furthermore, it surpasses all supervised fusion baselines in multimodal forecasting on Time-MMD, with Stage 2 achieving 0.514 NMAE. This demonstrates that a shared backbone can effectively handle both modalities without catastrophic interference, despite seeing significantly fewer training tokens (~138B text, ~12B TS patches) than text-only baselines.

Key takeaway

For AI Scientists and Machine Learning Engineers developing multimodal models for text and time series, you should reconsider the common practice of adapting pretrained language models. Chronicle demonstrates that training a compact 324M-parameter transformer from scratch on both modalities can achieve strong performance, matching LLMs on NLU and setting new benchmarks for time series tasks. This approach offers a more direct path to general-purpose representations, potentially leading to more efficient and effective models tailored for joint understanding.

Key insights

Jointly pretraining a single transformer from scratch on text and time series yields competitive multimodal representations.

Principles

• A shared transformer backbone can accommodate both text and time series effectively.
• Cross-modal capabilities can emerge from shared parameters via unimodal pretraining batches.
• Rigorous evaluation against unimodal foundation models is essential for multimodal claims.

Method

Train a decoder-only transformer from scratch on text tokens and time series patches, primarily using unimodal batches, followed by a short interleaved alignment stage. Modality-specific components are limited to input/output interfaces.

In practice

• For short time series, repeat TS-token blocks (e.g., r=64) to rebalance the modality ratio.
• Benchmark multimodal models against dedicated unimodal foundation models in each domain.

Topics

• Multimodal Foundation Models
• Time Series Forecasting
• Natural Language Understanding
• Transformer Architectures
• Representation Learning
• Joint Pretraining

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Towards a Unified Generative Model for Scarce Time Series with Domain Experts — TimeMoDE unifies Diffusion Transformers and Mixture-of-Experts for robust time series generation in data-scarce settings.
• Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages — Multimodal LLMs must integrate text, speech, and vision for low-resource languages, moving beyond English-centric benchmarks.
• Semantics-Enhanced Retrieval-Augmented Time Series Forecasting — Time series forecasting improves by combining numerical similarity with semantic context from textual descriptions.
• Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification — A single discrete visual tokenizer unifies multimodal understanding and generation in autoregressive models.
• TRACE: A Temporal Conditional Estimation for Multimodal Time Series Foundation Models — TRACE systematically infers missing time series modalities from available data, enhancing robustness and cross-modal representation learning.
• SMGFM: Spectral Multimodal Graph Pretraining for Multimodal-Attributed Graphs — Spectral decomposition disentangles structural and modality-specific semantics in multimodal graphs for improved learning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.