DUALVISION: RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning

2026-04-22 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Computer Vision · Depth: Expert, extended

Summary

DualVision is a novel, lightweight fusion module designed to enhance the robustness of Multimodal Large Language Models (MLLMs) in visual reasoning tasks by integrating both RGB and infrared (IR) imagery. Traditional MLLMs, which primarily rely on RGB data, often fail under adverse conditions like fog, blur, or low light. DualVision addresses this by employing patch-level localized cross-attention to efficiently combine IR and RGB information, reducing computational overhead by approximately 75% compared to naive fusion. To facilitate its development and evaluation, the researchers also introduced two new datasets: DV-204K, comprising around 25K aligned IR-RGB image pairs with 204K modality-specific QA annotations for instruction tuning, and DV-500, a benchmark of 500 IR-RGB image pairs with 500 QA pairs for evaluating cross-modal reasoning under various visual degradations. Benchmarking against open and closed-source MLLMs demonstrated DualVision's superior empirical performance and improved robustness.

Key takeaway

Research Scientists developing MLLMs for real-world applications like autonomous driving should consider integrating infrared data with RGB inputs. DualVision's lightweight, multi-scale localized cross-attention fusion module offers a proven method to significantly improve model robustness under common visual degradations, such as fog or low light, while maintaining computational efficiency. You should explore similar fusion architectures and degradation-aware training protocols to enhance the reliability of your MLLMs in challenging environments.

Key insights

Integrating infrared and RGB data via localized cross-attention significantly boosts MLLM robustness in degraded visual conditions.

Principles

• IR complements RGB for robust perception.
• Localized cross-attention reduces computational overhead.
• Degradation-aware training improves model resilience.

Method

DualVision uses multi-scale localized cross-attention to fuse RGB and IR patch tokens, allowing RGB tokens to attend only to spatially corresponding IR regions. This hierarchical approach, with progressively expanding local attention radii, creates a unified IR-RGB representation.

In practice

• Use IR-RGB fusion for MLLMs in low-visibility scenarios.
• Employ localized cross-attention to optimize fusion compute.
• Train MLLMs with degraded inputs to enhance robustness.

Topics

• DualVision
• RGB-Infrared Fusion
• Multimodal Large Language Models
• Visual Degradation Robustness
• DV-204K Dataset

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• FusionRS: A Large-Scale RGB-Infrared Remote Sensing Dataset for Dual-Modal Vision-Language Foundation Models — FusionRS enables dual-modal RGB-infrared vision-language models through a novel dataset with IR-aware captions.
• SpaceDG: Benchmarking Spatial Intelligence under Visual Degradation — Visual degradations consistently and substantially impair MLLM spatial reasoning, but degradation-aware training can significantly improve robustness.
• DREAM: Extending Vision-Language Models with Dual-Objective Encoding for Cross-Modal Retrieval — Dual-objective encoding and hierarchical attention significantly enhance cross-modal video retrieval performance.
• Seeing Together:Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models — Multi-robot egocentric spatial reasoning benefits from physics-informed MLLMs and specialized cooperative datasets.
• PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models — Multimodal diffusion language models enable efficient parallel visual perception, overcoming autoregressive MLLM limitations.
• What does RL improve for Visual Reasoning? A Frankenstein-Style Analysis — RL primarily refines mid-to-late VLM layers, enhancing vision-to-reasoning alignment, not uniform visual perception.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.