Not All Patches are Equal: Sampling Matters for Visible-Infrared Pre-Training

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

Visible-infrared (VIS-IR) alignment is a key pre-training task for robust multi-sensor perception. Existing methods often use uniform patch-wise contrastive learning, which is unreliable due to imaging-physics differences making some spatially paired regions less comparable. This paper introduces Importance-Aware Sampling (IAS), a plug-and-play framework that adjusts training emphasis based on patch reliability. IAS derives patch weights from infrared structural cues, learns a soft importance mask (optionally warm-started), and employs a patch curriculum learning strategy. It is compatible with both patch-/correlation-level (UNIV-style) and image-level (ImageBind-style) contrastive baselines. Extensive experiments on multiple VIS-IR benchmarks, including IR semantic segmentation, IR object detection, VIS semantic segmentation, and cross-modal retrieval, demonstrate consistent improvements. For example, IAS (UNIV-style) achieved 52.32% mIoU on MFNet, 72.23% on SCUTSEG, and 75.85% on MSRS, improving over UNIV† by 1.18, 2.58, and 0.37 points, respectively. The learnable variant adds only 0.107M parameters and 0.0418 GFLOPs.

Key takeaway

For Machine Learning Engineers designing visible-infrared pre-training pipelines, you should integrate Importance-Aware Sampling (IAS) to enhance representation learning. Uniform patch sampling introduces noisy gradients from unreliable regions, hindering model performance. By weighting patches based on their structural importance, your models will achieve more stable optimization and superior transferability across tasks like semantic segmentation and object detection. Consider implementing a learnable sampling module with a Sobel-guided warm-up for adaptive importance refinement.

Key insights

Not all VIS-IR patches are equally alignable, and weighting them by importance improves representation learning.

Principles

• Uniform patch sampling is suboptimal for VIS-IR alignment due to varying patch reliability.
• Infrared structural cues (e.g., Sobel, HOG) correlate with patch alignability across modalities.
• Soft importance weighting of patches is superior to hard truncation for smoother optimization.

Method

Importance-Aware Sampling (IAS) derives patch weights from infrared structural cues, optionally learns a soft importance mask with a lightweight sampler (warm-started), and uses a patch curriculum to gradually expand from high-reliability to harder patches, reweighting contrastive loss objectives.

In practice

• Use Sobel or HOG responses to generate initial importance weights for infrared patches.
• Integrate importance weighting as per-patch coefficients into existing contrastive loss functions.
• Employ a curriculum learning schedule to stabilize early-stage optimization by prioritizing reliable patches.

Topics

• Visible-Infrared Alignment
• Contrastive Learning
• Importance Sampling
• Multi-sensor Perception
• Semantic Segmentation
• Object Detection

Code references

• KlayMa527/IAS

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Not All Patches are Equal: Sampling Matters for Visible-Infrared Pre-Training — Importance-Aware Sampling (IAS) improves VIS-IR pre-training by adaptively weighting patches based on reliability, overcoming uniform sampling limitations.
• Any Resolution Any Geometry: From Multi-View To Multi-Patch — URGT refines high-resolution depth and normal maps using a multi-patch transformer with global coherence.
• Importance-Aware OBS Pruning for Diffusion Models — Content-aware pruning, using spatial importance maps, enables perceptually faithful compression of diffusion models at high ratios.
• Texture-Shape Bias Balancing for Robust Synthetic-to-Real Semantic Segmentation in Automotive NIR Imagery — A generative augmentation framework reduces the synthetic-to-real domain gap for automotive NIR semantic segmentation by balancing texture-shape bias.
• Dual-Integrated Low-Latency Single-Lens Infrared Computational Imaging for Object Detection — Integrating optical physics priors with deep learning for joint reconstruction-detection enables compact, low-latency infrared object detection.
• Spectral Prior for Reducing Exposure Bias in Diffusion Models — Diffusion models' exposure bias stems from frequency-dependent SNR errors, correctable via spectral alignment guidance.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.