A foundation model of vision, audition, and language for in-silico neuroscience

2026-05-05 · Source: Takara TLDR - Daily AI Papers · Field: Science & Research — Artificial Intelligence & Machine Learning, Life Sciences & Biology, Health & Medical Research · Depth: Expert, medium

Summary

TRIBE v2 is a new tri-modal foundation model integrating video, audio, and language to predict human brain activity, addressing the fragmentation of specialized models in cognitive neuroscience. Developed by Jean-Rémi King and colleagues, the model leverages a unified dataset of over 1,000 hours of fMRI data from 720 subjects. TRIBE v2 accurately predicts high-resolution brain responses for novel stimuli, tasks, and subjects, significantly outperforming traditional linear encoding models with several-fold improvements in accuracy. The model also enables "in silico" experimentation, successfully recovering results from seminal visual and neuro-linguistic paradigms established over decades of empirical research. Furthermore, TRIBE v2 extracts interpretable latent features, revealing the fine-grained topography of multisensory integration, positioning AI as a unifying framework for understanding brain functional organization.

Key takeaway

For AI Scientists and Research Scientists developing brain-computer interfaces or cognitive models, TRIBE v2 demonstrates a powerful approach to unifying multimodal data for brain activity prediction. You should consider adopting foundation models that integrate diverse sensory inputs to achieve more generalized and accurate representations of brain function, potentially accelerating in silico experimentation and the discovery of multisensory integration mechanisms.

Key insights

TRIBE v2 unifies vision, audition, and language to predict human brain activity, enabling in silico neuroscience.

Principles

• Unified models overcome fragmented cognitive neuroscience.
• Foundation models can predict novel brain responses.
• In silico experiments validate empirical research.

Method

TRIBE v2 uses a tri-modal foundation model trained on over 1,000 hours of fMRI data across 720 subjects to predict high-resolution brain activity and extract interpretable latent features.

In practice

• Predict brain responses to new stimuli.
• Conduct virtual neuro-linguistic experiments.
• Map multisensory integration topography.

Topics

• TRIBE v2
• Foundation Models
• In-silico Neuroscience
• Brain Activity Prediction
• Multisensory Integration

Best for: AI Scientist, Research Scientist

Related on AIssential

• A foundation model of vision, audition, and language for in-silico neuroscience | Research - AI at Meta — TRIBE v2 unifies cognitive neuroscience by predicting human brain activity across diverse conditions using a tri-modal foundation model.
• Introducing TRIBE v2: A Predictive Foundation Model Trained to Understand How the Human Brain Processes Complex Stimuli - AI at Meta — TRIBE v2 is a foundation model predicting high-resolution human brain activity from diverse sensory stimuli.
• Today we're introducing TRIBE v2 (Trimodal Brain Encoder), a foundation model trained to predict how the human brain responds to almost any sight or sound. Building on our Algonauts 2025 award-winning architecture, TRIBE v2 draws on 500+ hours of fMR - x.com — TRIBE v2 is a foundation model predicting brain responses to sights and sounds using fMRI data.
• Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion — Mind-Omni unifies brain-vision-language tasks via discrete diffusion and a Brain Tokenizer for multi-modal interaction.
• fMRI-LM: Towards a Universal Foundation Model for Language-Aligned fMRI Understanding — fMRI-LM aligns fMRI with LLMs using synthetic text descriptors, enabling unified, language-grounded brain activity understanding.
• Boosting Brain-to-Image Decoding with TRIBE v2 Data Augmentation — Synthetic fMRI data from pretrained models like TRIBE v2 can significantly boost brain-to-image decoding in low-data regimes.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.