Beyond Parallel Tracking: Interactive Multi-Feature Fusion Drives Semantic Reconstruction from Non-invasive Brain Recordings

2026-07-15 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Neuroscience & Brain-Computer Interfaces · Depth: Expert, extended

Summary

This study introduces a multi-feature fusion framework for non-invasive semantic reconstruction, addressing the representational mismatch between semantic features and neural coding. It systematically benchmarks two integration approaches: linear Naive Concatenation and non-linear Multi-Head Cross-Attention. The framework combines static lexical representations (Word2Vec) with dynamic contextual representations (GPT) using an interactive gating mechanism. Evaluated on the SEM4Lang MEG dataset, comprising 60.7 hours from 12 Chinese speakers, the framework demonstrated a robust performance hierarchy: Cross-Att > Concat > GPT > W2V. The non-linear cross-attention fusion achieved superior performance in semantic reconstruction (e.g., higher Top-10 accuracy with longer temporal windows) and text generation (BLEU-1 score of 15.58±1.16, METEOR score of 9.23±0.89 with ConvConcatNet). This indicates that simulating collaborative modulation between contextual information and core lexical attributes significantly improves non-invasive brain-to-text decoding over single-feature methods.

Key takeaway

For Machine Learning Engineers developing non-invasive brain-to-text decoders, you should integrate both static lexical and dynamic contextual semantic features. Prioritize non-linear Multi-Head Cross-Attention for feature fusion, as it significantly outperforms single-feature or linear concatenation methods. This approach enhances decoding accuracy and text generation quality from noisy neural signals. Your models will achieve superior cross-modal alignment and reduce overfitting by constraining the hypothesis space, moving closer to practical BCI deployment.

Key insights

Non-invasive brain-to-text decoding improves significantly by interactively fusing static lexical and dynamic contextual features.

Principles

• Human language processing integrates static and dynamic features.
• Multi-feature fusion reduces information loss in neural decoding.
• Non-linear cross-attention outperforms linear concatenation for feature integration.

Method

The framework uses Word2Vec and GPT for static/dynamic features, then fuses them via Multi-Head Cross-Attention. Self-supervised contrastive learning aligns neural signals with these fused semantic targets for text generation.

In practice

• Combine W2V and GPT embeddings for robust semantic targets.
• Implement cross-attention for non-linear feature integration.
• Use InfoNCE loss for neural signal-to-semantic target alignment.

Topics

• Non-invasive Brain-Computer Interfaces
• Semantic Reconstruction
• Multi-Head Cross-Attention
• MEG Signal Processing
• Language Model Embeddings
• Brain-to-Text Decoding

Best for: AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• Beyond Parallel Tracking: Interactive Multi-Feature Fusion Drives Semantic Reconstruction from Non-invasive Brain Recordings — Interactive multi-feature fusion of static and dynamic semantic representations significantly improves non-invasive brain-to-text decoding.
• Audio-Text Cross-Attention with Psycholinguistic Support Features for Ambivalence/Hesitancy Recognition — Combining audio, text, and psycholinguistic features via cross-attention improves ambivalence recognition.
• Non-literal Meaning Representation in the Brain during Naturalistic Listening — The brain processes literal and non-literal language meaning in distinct yet complementary neural regions.
• Revisiting Structural Dependency in Autoregressive Multi-Task Table Recognition via Order-Independent Cell-Level Representations — Order-independent cell representations, achieved via non-causal attention, enhance multi-task table recognition and reduce inference time.
• Decoding EEG Signals to Explore Next-Word Predictability in the Human Brain — EEG analysis reveals distinct N400 responses to next-word predictability across lexical categories, with decoding outperforming traditional ERP methods.
• Contextual Semantic Relevance and Word Surprisal Predict N400 and P600 Dynamics During Naturalistic Reading — Naturalistic reading simultaneously processes lexical expectation and semantic integration, each with distinct neural correlates.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.