MonoIR-RS: Infrared Remote Sensing Vision-Language Learning with CLIP and VLM Adaptation

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Computer Vision for Remote Sensing · Depth: Expert, extended

Summary

MonoIR-RS, a large-scale infrared remote-sensing vision-language dataset and benchmark, addresses the underexplored area of infrared vision-language understanding, which is often overlooked by models focused on visible-band semantics. Built from the FusionRS source pool, MonoIR-RS comprises 600,000 synthesized infrared images and 59,032 IR-aware caption records, retaining only the infrared image as the model-facing modality. The synthetic infrared imagery is validated to be markedly closer to real thermal imagery than a grayscale conversion on the AVIID benchmark, with an FID of 85.2 versus 126.3. The study fine-tunes five CLIP backbones and six VLM backbones, demonstrating that IR-aware adaptation boosts CLIP mean recall by up to +12.8 points (best checkpoint 19.2% on the 9,720-image filtered split) and drives VLM captioning IR-cue coverage to 100% while reducing RGB-color leakage to near zero. This work offers a controlled, reproducible testbed for aligning infrared remote-sensing evidence with language.

Key takeaway

For machine learning engineers developing vision-language models for infrared remote sensing, you should prioritize IR-aware data and adaptation. Relying on RGB-centric models or captions will lead to poor performance due to modality mismatch. Instead, utilize synthetic infrared imagery and rewrite supervision to emphasize grayscale structure and thermal contrast. This approach significantly improves retrieval accuracy and ensures generative models produce infrared-grounded descriptions, avoiding color leakage.

Key insights

Infrared remote sensing vision-language models require IR-aware data and adaptation to overcome visible-band biases.

Principles

• Infrared imagery needs captions emphasizing grayscale structure, not RGB appearance.
• Synthetic infrared data can effectively bridge the modality gap for vision-language model training.
• Separate evaluation for retrieval and generative models reveals distinct failure modes.

Method

MonoIR-RS synthesizes 600,000 infrared images from visible sources via DiffV2IR and rewrites 59,032 captions with Qwen2.5-VL-72B-Instruct to emphasize IR-style evidence. It fine-tunes CLIP and VLM backbones with train-only data.

In practice

• Utilize DiffV2IR for visible-to-infrared image synthesis.
• Rewrite captions to focus on grayscale structure and thermal contrast.
• Evaluate VLMs with lexical diagnostics for IR-cue rate and color leakage.

Topics

• Infrared Remote Sensing
• Vision-Language Models
• CLIP Adaptation
• VLM Instruction Tuning
• Synthetic Data Generation
• Dataset Benchmarking

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• From Machine Translation to Image Captioning: Training Vision-Language Models for Indigenous Languages of the Americas — Intermediate machine translation fine-tuning significantly improves vision-language models for Indigenous language image captioning.
• Vision-Language Guided Hyperspectral Object Tracking via Semantics Fusion and Contextual Template Updating — VLHTrack integrates vision and language with dynamic template updates to enhance hyperspectral object tracking performance.
• FusionRS: A Large-Scale RGB-Infrared Remote Sensing Dataset for Dual-Modal Vision-Language Foundation Models — FusionRS enables dual-modal RGB-infrared vision-language models through a novel dataset with IR-aware captions.
• Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models — SPpruner reduces VLM computational cost by mimicking human "Focus-then-Context" perception to retain salient subjects and their structural context.
• TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment — TEVI enhances vision-language alignment by using captions to selectively edit image embeddings via sparse autoencoders.
• Textual Supervision Enhances Geospatial Representations in Vision-Language Models — Textual supervision in VLMs significantly improves implicit geospatial representation learning compared to vision-only models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.