CausalMoE: A Billion-Scale Multimodal Foundation Model for Granger Causal Discovery with Pattern-Routed Heterogeneous Experts

2026-06-12 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

CausalMoE is a billion-scale multimodal foundation model designed for Granger Causal Discovery (GCD) in time series, addressing the limitations of "one-size-fits-all" neural methods that struggle with distribution shifts. It introduces a Pattern-Routed Mixture of Heterogeneous Experts (MoHE) to dynamically identify latent temporal patterns and route time-series patches to specialized domain experts, decoupling regime-specific mechanisms. CausalMoE integrates Large Language Models (LLMs) and Vision-Language Models (VLMs) to align numerical signals with textual and visual priors, regularizing causal estimation. A Causality-Aware Self-Attention mechanism ensures interpretable, sparse graph recovery via proximal optimization. Extensive experiments show CausalMoE achieves state-of-the-art performance on benchmarks like VAR, Lorenz-96, fMRI, DREAM-3, and DREAM-4, demonstrating strong generalization, especially in few-shot settings where traditional methods fail.

Key takeaway

For Machine Learning Engineers developing causal inference systems, CausalMoE offers a robust solution for Granger Causal Discovery, particularly in data-scarce or heterogeneous time series environments. Its ability to leverage multimodal priors and adapt to regime shifts means you can achieve reliable causal structures with significantly less training data, outperforming traditional methods. Consider integrating this approach to enhance the accuracy and interpretability of your temporal causal models.

Key insights

CausalMoE uses multimodal experts and pattern-routed architecture for robust Granger Causal Discovery in heterogeneous time series.

Principles

• Explicitly model patch-level heterogeneity for reliable causal discovery.
• Multimodal priors from LLMs/VLMs regularize causal estimation.
• Causality-Aware Self-Attention yields sparse, interpretable causal graphs.

Method

CausalMoE employs Multimodal Patch Encoding, Patch-Specific Pattern Routing to heterogeneous experts (Semantic, Multimodal, Temporal Frequency, Multiscale Temporal), and Causality-Aware Self-Attention with proximal optimization for sparse graph recovery.

In practice

• Integrate LLMs and VLMs to enrich time series representations.
• Apply Mixture of Experts for adaptive modeling of temporal heterogeneity.
• Use variable-wise attention for direct causal interpretation.

Topics

• Granger Causal Discovery
• Multimodal Foundation Models
• Time Series Analysis
• Mixture-of-Experts
• Large Language Models
• Vision-Language Models

Code references

• liubolab/CausalMoE

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• CausalMoE: A Billion-Scale Multimodal Foundation Model for Granger Causal Discovery with Pattern-Routed Heterogeneous Experts — CausalMoE uses pattern-routed heterogeneous experts and multimodal priors for robust Granger Causal Discovery in complex, dynamic time series.
• Predicting Causal Effects from Natural Language Queries using Structured Representations — Finetuning and structured representations significantly enhance LLM performance in predicting causal effects from natural language.
• Causal Foundation Models with Continuous Treatments — A novel causal foundation model meta-learns continuous treatment effects, outperforming specialized models via in-context learning.
• TRACE: A Temporal Conditional Estimation for Multimodal Time Series Foundation Models — TRACE leverages conditional diffusion for robust multimodal time series imputation, outperforming naive methods under missingness.
• From Prompts to Tokens: Internalizing Causal Supervision in Vision-Language Model for Multi-Image Causal Reasoning — BridgeVLM internalizes visual causal reasoning in VLMs by converting causal graphs into executable tokens for improved control.
• Causal Interventions on Continuous Variables: A Case Study on Verb Bias in Steering Vectors for In-Context Learning — Causal interventions can effectively manipulate continuous variables like verb bias in language model steering vectors.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.