LoMo: Local Modality Substitution for Deeper Vision-Language Fusion

2026-05-28 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Local Modality Substitution (LoMo) is a lightweight, architecture-agnostic data curation paradigm designed to address the "carrier sensitivity" issue in Vision-Language Models (VLMs). This problem causes significant performance degradation when textual questions are replaced by their rendered-image counterparts, stemming from an inherent bias in training corpora where text and images typically serve distinct roles. LoMo tackles this by reformulating single-modality prompts into seamlessly interleaved multimodal sequences, dynamically selecting text spans and recasting them as rendered images. This process preserves semantic equivalence across "text, visual, text" carriers. Extensive experiments across 13 diverse multimodal benchmarks show LoMo consistently improves multimodal reasoning and cross-modal fusion, delivering gains of 2.67 points on LLaVA-OneVision-1.5-8B and 2.82 points on Qwen3.5-9B over standard SFT.

Key takeaway

For machine learning engineers developing robust Vision-Language Models, LoMo presents a lightweight, architecture-agnostic data curation paradigm to mitigate "carrier sensitivity." You should consider integrating this approach into your VLM training pipelines to achieve deeper cross-modal fusion and consistent performance gains, as demonstrated on models like LLaVA-OneVision-1.5-8B and Qwen3.5-9B. This can significantly enhance your model's reasoning capabilities under modality substitution.

Key insights

LoMo addresses VLM "carrier sensitivity" by creating semantically equivalent text and image representations through local modality substitution.

Principles

• VLMs exhibit "carrier sensitivity" due to data bias in training corpora.
• Cross-modal representational invariance is crucial for robust VLM reasoning.

Method

LoMo reformulates single-modality prompts into interleaved multimodal sequences, dynamically selecting text spans and recasting them as rendered images to ensure semantic equivalence across modalities.

In practice

• Apply LoMo to improve VLM performance on diverse multimodal benchmarks.
• Use LoMo with foundational models like LLaVA-OneVision-1.5-8B and Qwen3.5-9B.

Topics

• Vision-Language Models
• Modality Substitution
• Cross-modal Fusion
• Data Curation
• Carrier Sensitivity
• LLaVA-OneVision

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention — Faithful-MR1 improves MLLM reasoning by directly anchoring visual attention and reinforcing its causal use.
• Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages — Multimodal LLMs must integrate text, speech, and vision for low-resource languages, moving beyond English-centric benchmarks.
• xModel-KD: Cross-modal Knowledge Distillation for 3D Scene Perception using LiDAR — Cross-modal knowledge distillation improves 3D point cloud segmentation by fusing 2D texture and 3D geometry for richer representations.
• Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models — Loc3R-VLM enhances 2D VLMs with 3D understanding via joint global layout reconstruction and egocentric situation modeling.
• Report of the 5th PVUW Challenge: Towards More Diverse Modalities in Pixel-Level Understanding — Multimodal foundation models are crucial for robust pixel-level video understanding in complex, unconstrained environments.
• What is Multimodal AI? How LLMs Process Text, Images, and More — Native multimodality uses a shared vector space for unified processing of diverse data types, enabling richer AI reasoning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.