A Good Initialization is All You Need for Faithful Visual Attribution

2026-07-07 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Faithful visual attribution, which identifies image regions supporting model predictions, is advanced by new search-based perturbation methods. While existing techniques often output complete region orderings, many applications, particularly MLLM attribution and repair, require compact top-k evidence masks. This work introduces two forward-only methods: CoPAIR and TRACE. CoPAIR uses a PhaseWin-Greedy gap diagnosis to create coarse singleton/pair candidates for full-ordering search. TRACE directly searches fixed-cardinality fine-region masks via cross-entropy sampling, elite retention, and distribution updates. These initialized search methods establish a new state-of-the-art for faithful full-ordering attribution on ImageNet with CLIP ViT-L/14, CLIP RN101, and ResNet-101. TRACE+Greedy yields the strongest search-based MLLM attribution on POPE and RePOPE with Qwen2.5-VL-3B-Instruct and LLaVA-v1.5-7B, with direct TRACE masks achieving 94.44% and 96.00% RePOPE repair rates.

Key takeaway

For Machine Learning Engineers and AI Scientists developing visual attribution systems or working on MLLM interpretability, you should consider integrating CoPAIR or TRACE. These methods provide superior faithfulness for both full-ordering and compact top-k evidence masks, significantly improving MLLM attribution results and achieving high repair rates. Implementing these initialized search techniques can enhance the actionability and reliability of your model explanations.

Key insights

Effective initialization significantly enhances the faithfulness and compactness of visual attribution masks.

Principles

• Useful evidence can depend on interactions among fine regions.
• Coarse grouping can stabilize early search but aggregates redundant content.

Method

CoPAIR constructs coarse singleton/pair candidates using PhaseWin-Greedy gap diagnosis to warm-start full-ordering search. TRACE directly searches fixed-cardinality fine-region masks via cross-entropy sampling, elite retention, and distribution updates.

In practice

• Use TRACE for compact top-k attribution masks.
• Initialize Greedy or PhaseWin with TRACE for complete rankings.
• Apply TRACE masks for MLLM attribution and repair tasks.

Topics

• Visual Attribution
• Multimodal LLMs
• Model Interpretability
• CLIP ViT-L/14
• Qwen2.5-VL-3B-Instruct
• TRACE Algorithm

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• PhaseWin: An Efficient Search Algorithm for Faithful Visual Attribution — PhaseWin efficiently achieves faithful visual attribution by optimizing greedy search with a phased window-search procedure.
• Influcoder: Distilling Decoders' Gradient Influence Rankings into an Encoder for Data Attribution — Influcoder enables scalable, cost-effective data attribution for LLMs by distilling decoder influence into an encoder.
• LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data — LLMs struggle with epistemic self-awareness on structured clinical data, but cross-model attribution divergence can detect blind spots and improve reliability.
• Forecasting Is Not Attribution: Localizing Decoder Bypass in Graph-Based Neural Marketing Mix Models — Forecasting accuracy in neural MMMs does not guarantee correct attribution, often hiding a "decoder bypass" failure.
• LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models — LOCUS enhances MLLM fine-grained perception by internalizing local visual cue search via a training-time proxy task.
• CAVE: A Structured Credit Assignment Approach for Fragmented Visual Evidence Reasoning — CAVE improves VLM reasoning by assigning structured, action-level credits for visual evidence exploration.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.