AIM: Asymmetric Information Masking for Visual Question Answering Continual Learning

2026-04-16 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new method called Asymmetric Information Masking (AIM) has been developed to address catastrophic forgetting in continual Visual Question Answering (VQA) for modern Vision-Language Models (VLMs). Existing Continual Learning (CL) methods are designed for symmetric, unimodal architectures, which conflicts with the inherently asymmetric trainable components of VLMs. This mismatch causes standard global regularization to prioritize the large language decoder, leaving the smaller visual projection layers susceptible to interference and leading to a loss of compositional reasoning. AIM mitigates this by applying targeted masks based on modality-specific sensitivity to balance stability and plasticity. Experiments on VQA v2 and GQA datasets demonstrate that AIM achieves state-of-the-art performance in Average Performance (AP) and Average Forgetting (AF), while also improving generalization to novel skill-concept compositions.

Key takeaway

For research scientists developing continual learning strategies for Vision-Language Models, AIM offers a critical solution to catastrophic forgetting. By addressing the inherent architectural asymmetry, AIM significantly improves performance and generalization on VQA tasks. You should consider integrating AIM's modality-specific masking approach to enhance the stability and plasticity of your VLM training pipelines, particularly when dealing with continuous data streams.

Key insights

Asymmetric Information Masking (AIM) prevents catastrophic forgetting in VQA by balancing VLM stability and plasticity.

Principles

• VLM asymmetry causes catastrophic forgetting.
• Modality-specific masking balances stability and plasticity.

Method

AIM applies targeted masks based on modality-specific sensitivity to balance stability and plasticity in asymmetric VLM architectures during continual learning.

In practice

• Apply AIM to VQA models for continual learning.
• Use AIM to improve generalization in VLMs.

Topics

• Visual Question Answering
• Continual Learning
• Vision-Language Models
• Asymmetric Information Masking
• Catastrophic Forgetting

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs — Visual latents in MLLMs can be "silenced" by training objectives, but their reasoning capacity can be unleashed at inference time.
• StrLoRA: Towards Streaming Continual Visual Instruction Tuning for MLLMs — Textual instructions provide a stable signal for expert routing in streaming multimodal learning.
• Noise-Aware Visual Representation Learning for Medical Visual Question Answering — Denoising visual embeddings before LLM projection significantly improves Med-VQA robustness against input noise.
• The Expense of Seeing: Attaining Trustworthy Multimodal Reasoning Within the Monolithic Paradigm — Current VLMs often exhibit "functional blindness," relying on language priors over true visual understanding.
• MAny: Merge Anything for Multimodal Continual Instruction Tuning — MAny mitigates dual-forgetting in MLLMs via training-free merging of perceptual and reasoning knowledge.
• Diagnosing Visual Ignorance in Vision-Language Models — Vision-Language Models frequently prioritize language priors over visual evidence due to internal routing failures and flawed benchmark evaluations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.