What Does the Caption Really Say? Counterfactual Phrase Intervention for Compositional Data Selection in Vision-Language Pretraining

2026-05-21 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision, Natural Language Processing · Depth: Expert, quick

Summary

Counterfactual Phrase Intervention (CPI) is a novel phrase-level curation framework for Vision-Language Pretraining. It addresses limitations of traditional CLIP-style pair-level filtering. Existing methods, relying on global alignment scores, often saturate and fail to capture whether individual caption phrases truly support the image-text match. CPI converts controlled nonce-token substitutions into image-conditioned phrase-sensitivity scores. It first removes coarse mismatches using global alignment. Then, it ranks remaining data by how measurably caption phrases affect the image-text score under substitution. Evaluated at CC3M scale, CPI yields a 50%-data subset. This improves VL-CheckList-VG Relation by +1.91 over the full-data baseline and +1.00 over alignment-only filtering. It also improves SugarCrepe and preserves general transfer, demonstrating effectiveness and loss-orthogonality, with NegCLIP gains of +3.84.

Key takeaway

For Machine Learning Engineers curating datasets for vision-language pretraining, you should consider integrating phrase-level data selection methods like Counterfactual Phrase Intervention (CPI). Relying solely on global alignment scores can limit compositional generalization. CPI offers a more granular approach, improving metrics like VL-CheckList-VG Relation by +1.91 on 50% data. This enables building more robust models with better compositional understanding. It also applies to existing architectures like NegCLIP, yielding +3.84 gains.

Key insights

Counterfactual Phrase Intervention uses phrase-level sensitivity to select compositional data, outperforming global alignment in vision-language pretraining.

Principles

• Global alignment saturates for compositional supervision.
• Phrase-level sensitivity tracks compositional support.
• Controlled substitution reveals phrase impact.

Method

CPI first removes coarse mismatches via global alignment. It then ranks the remaining data by image-conditioned phrase-sensitivity scores, generated from controlled nonce-token substitutions, to measure individual phrase impact on image-text alignment.

In practice

• Apply CPI to existing CLIP-style models.
• Curate vision-language datasets for compositional tasks.
• Improve compositional generalization via phrase filtering.

Topics

• Vision-Language Pretraining
• Data Curation
• Counterfactual Phrase Intervention
• Compositional Generalization
• CLIP
• NegCLIP

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• When Vision Speaks for Sound — MLLMs often hallucinate audio from visual cues, exhibiting a "Clever Hans effect" instead of true audio-visual grounding.
• CoCoGEC: Counterfactual Generation for Robust Grammatical Error Correction — Training GEC models with context-altered counterfactuals significantly enhances their robustness against contextual perturbations.
• Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field — Consistent evaluation reveals many reported SLT performance gains are artifacts of inconsistent implementation and evaluation setups.
• Diagnosing Visual Ignorance in Vision-Language Models — Vision-Language Models frequently prioritize language priors over visual evidence due to internal routing failures and flawed benchmark evaluations.
• Task-Instructed Causal Routing of Vision Foundation Models for Multi-Task Learning — TIGER coordinates heterogeneous VFMs for multi-task dense prediction using instruction-guided routing and counterfactual causal alignment.
• G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignment — Cross-lingual idiom alignment challenges LLMs due to non-compositionality, with glosses offering semantic grounding but literal bias remaining dominant.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.