Seeing the imagined: a latent functional alignment in visual imagery decoding from fMRI data

2026-04-20 · Source: cs.AI updates on arXiv.org · Field: Science & Research — Artificial Intelligence & Machine Learning, Health & Medical Research · Depth: Expert, extended

Summary

Researchers Fabrizio Spera, Tommaso Boccato, and colleagues investigated adapting a state-of-the-art visual perception decoder, DynaDiff, to reconstruct imagined content from fMRI data using the Imagery-NSD benchmark. They introduced a latent functional alignment approach that maps imagery-evoked brain activity into the pretrained model's conditioning space, keeping other components frozen. To address limited matched imagery-perception supervision, a retrieval-based augmentation strategy was developed, selecting semantically related Natural Scenes Dataset (NSD) perception trials. Across four subjects, this latent functional alignment consistently improved high-level semantic reconstruction metrics compared to a frozen pretrained baseline and a voxel-space ridge alignment baseline. The method enabled above-chance decoding from multiple cortical regions, suggesting that semantic structures learned from perception can stabilize and enhance visual imagery decoding under out-of-distribution conditions.

Key takeaway

For AI Scientists and Machine Learning Engineers developing brain-computer interfaces or fMRI decoding models, consider implementing latent functional alignment and retrieval-based data augmentation. This approach significantly enhances semantic reconstruction of imagined content, particularly when working with limited imagery datasets and adapting models initially trained on visual perception. Your focus should be on high-level semantic metrics to accurately assess performance, as pixel-level fidelity may be less critical for internally generated mental images.

Key insights

Latent functional alignment significantly improves fMRI-based visual imagery decoding by leveraging perception-learned semantic structures.

Principles

• Semantic structure from perception aids imagery decoding.
• Imagery decoding benefits from latent functional alignment.
• Distributed cortical networks contribute to visual imagery.

Method

A latent functional alignment approach maps imagery-evoked fMRI activity into a pretrained model's conditioning space. A retrieval-based augmentation strategy selects semantically related perception trials to mitigate data scarcity.

In practice

• Adapt perception decoders for imagery tasks.
• Augment imagery datasets with semantically similar perception data.
• Focus on high-level semantic metrics for imagery reconstruction.

Topics

• fMRI Decoding
• Visual Imagery Reconstruction
• Latent Functional Alignment
• DynaDiff Model
• Natural Scenes Dataset

Best for: AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• Beyond Visual Memory: Mechanistic Diagnostics of Latent Visual Reasoning — Gains in latent visual reasoning stem from boundary markers and format, not visual memory encoded in latent slots.
• Achieving more human brain-like vision via human EEG representational alignment — ReAlnet aligns AI vision models with human EEG data for more brain-like object recognition.
• DREAM: Extending Vision-Language Models with Dual-Objective Encoding for Cross-Modal Retrieval — Dual-objective encoding and hierarchical attention significantly enhance cross-modal video retrieval performance.
• Modeling visual memorability assessment with autoencoders reveals characteristics of memorable images — Autoencoder-based models can identify visual features that predict image memorability.
• Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection — Structured latent projection in multimodal segmentation improves robustness and preserves complementary information under varying modality availability.
• Neural Visual Decoding via Cognitive guided Adaptive Blurring and Information Constrained Alignment — CAIA enhances EEG-based visual decoding by adaptively blurring images and filtering EEG signals based on cognitive principles.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.