FusionRS: A Large-Scale RGB-Infrared Remote Sensing Dataset for Dual-Modal Vision-Language Foundation Models

2026-06-15 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Remote Sensing · Depth: Expert, quick

Summary

FusionRS is introduced as the first large-scale RGB-infrared-text dataset designed for dual-modal vision-language learning in remote sensing, addressing a significant gap in Earth observation understanding. While most existing remote sensing vision-language models focus on RGB imagery, FusionRS incorporates infrared data, which offers unique cues like thermal intensity structures and illumination-invariant features. The dataset is constructed by translating public RGB remote sensing images into infrared-style counterparts, creating aligned RGB-IR pairs. Each pair includes conventional scene captions and specific IR-aware captions that describe infrared properties while maintaining semantic content. Using FusionRS, researchers trained CLIP-style models for RGB-IR-text alignment and fine-tuned generative Vision-Language Models for dual-modal RGB-IR captioning. Experiments confirm that FusionRS enhances RGB-IR alignment, infrared-to-text retrieval, and dual-modal captioning compared to RGB-only or non-IR-aware training, with IR-aware captions proving vital for strengthening infrared-language alignment.

Key takeaway

For Machine Learning Engineers developing Earth observation models, FusionRS offers a critical resource to move beyond RGB-only approaches. You should consider leveraging this large-scale RGB-infrared-text dataset to train dual-modal vision-language foundation models. Integrating IR-aware captions, as demonstrated, significantly improves RGB-IR alignment and retrieval, enhancing your model's understanding of complex remote sensing data.

Key insights

FusionRS enables dual-modal RGB-infrared vision-language models through a novel dataset with IR-aware captions.

Principles

• Infrared data enriches remote sensing VLM beyond RGB.
• Modality-specific textual supervision strengthens alignment.
• Dual-modal datasets improve retrieval and captioning.

Method

FusionRS is constructed by translating public RGB remote sensing images into infrared-style counterparts, forming aligned RGB-IR pairs. Each pair is associated with conventional scene captions and IR-aware captions.

In practice

• Use FusionRS for training RGB-IR vision-language models.
• Incorporate IR-aware captions for better IR-language alignment.
• Explore synthetic IR data generation from RGB.

Topics

• Remote Sensing
• Vision-Language Models
• RGB-Infrared Data
• Dataset Generation
• Dual-Modal Learning
• Earth Observation

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• DUALVISION: RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning — Integrating infrared and RGB data via localized cross-attention significantly boosts MLLM robustness in degraded visual conditions.
• SyMTRS: Benchmark Multi-Task Synthetic Dataset for Depth, Domain Adaptation and Super-Resolution in Aerial Imagery — SyMTRS is a multi-task synthetic dataset for aerial imagery, providing perfect ground truth for depth, domain adaptation, and super-resolution.
• Unlocking Multi-Spectral Data for Multi-Modal Models with Guided Inputs and Chain-of-Thought Reasoning — Generalist LMMs can interpret multi-spectral data zero-shot by converting it to pseudo-images and using detailed, CoT-enhanced prompts.
• HQF-Net: A Hybrid Quantum-Classical Multi-Scale Fusion Network for Remote Sensing Image Segmentation — Hybrid quantum-classical networks can significantly improve remote sensing semantic segmentation by fusing multi-scale features and quantum-enhanced refinement.
• Emerging Flexible Designs for Geospatial Multimodal Foundation Models — Geospatial foundation model architecture choices, particularly fusion and tokenization strategies, critically impact flexibility and performance with varying spectral bands.
• Modality Forcing for Scalable Spatial Generation — Modality Forcing enables scalable, joint image-depth generation using sparse data and T2I models, outperforming prior methods.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.