Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality

2026-06-12 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Multimodal AI · Depth: Expert, extended

Summary

The MACCO (MAsked Compositional Concept MOdeling) framework addresses the compositional understanding limitations of contrastively trained vision-language models (VLMs) like CLIP, which often exhibit "bag-of-words" behavior. MACCO improves VLM compositionality by masking compositional concepts in one modality and reconstructing them conditioned on the full context from the other. This process is facilitated by two auxiliary objectives: Masked-augmented Cross-Modal Alignment Loss (MCA) and Masked-augmented Intra-Modal Regularization Loss (MIR), alongside a global-to-local semantic injection operation. Extensive experiments on five compositional benchmarks demonstrate MACCO's effectiveness, yielding significant improvements such as 14.4% on ARO-Relation and an 8.3% average gain on Sugar-Crepe. The enhanced compositionality also benefits downstream tasks like text-to-image generation and multimodal large language models. Code is publicly available.

Key takeaway

For AI Scientists and Machine Learning Engineers developing vision-language models, if your current CLIP-based models exhibit "bag-of-words" behavior and struggle with compositional understanding, you should consider integrating the MACCO framework. This method, which uses masked compositional concept modeling and auxiliary losses, significantly enhances attribute binding, relation understanding, and word order sensitivity. Implementing MACCO can improve your VLM's core reasoning capabilities and directly benefit downstream applications like text-to-image generation and multimodal large language models.

Key insights

Masking and reconstructing cross-modal compositional concepts improves VLM understanding of relations, attributes, and word order.

Principles

• VLMs often exhibit "bag-of-words" behavior.
• Masked cross-modal reconstruction enhances compositional understanding.
• Auxiliary losses regularize feature space and aid reconstruction.

Method

MACCO identifies compositional concepts in text (scene graph parser) and images (GroundingDINO), then masks them in one modality for reconstruction using the other's full context. It employs global-to-local semantic injection and optimizes with L_MLM, L_MIM, L_MCA, and L_MIR.

In practice

• Apply MACCO to improve text-to-image generation.
• Enhance MLLM visual backbones with MACCO-trained encoders.
• Combine MACCO with hard negative mining for further gains.

Topics

• Vision-Language Models
• Compositional Reasoning
• Masked Modeling
• CLIP
• Text-to-Image Generation
• Multimodal LLMs

Code references

• hiker-lw/MACCO

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality — MACCO improves VLM compositional understanding by cross-modal masked concept reconstruction and auxiliary alignment objectives.
• From Pixels to Prompts: Vision-Language Models — Understanding Vision-Language Models requires a clear mental map beyond just knowing buzzwords.
• Magnifying What Matters: Attention-Guided Adaptive Rendering for Visual Text Comprehension — VLMs localize text evidence without full utilization; adaptive rendering improves comprehension.
• Understanding Pruning Regimes in Vision-Language Models Through Domain-Aware Layer Selection — Domain-aware pruning effectively reduces VLM depth by identifying and removing layers least critical to specific tasks.
• Claude Fable 5 Drops (Beats Opus) — Multimodal language models unify diverse data streams by tokenizing all modalities for transformer-based auto-regressive generation.
• Language-Guided Abstraction for Visual Reasoning — L-VARC enhances visual reasoning for ARC tasks by integrating language priors through a lightweight, language-guided learning framework.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.