Sparse Attention for Dense Open-Vocabulary Prediction in CLIP

2026-07-08 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

Contrastive Language-Image Pre-training (CLIP) models, which typically use dense softmax-based self-attention, often spread attention across semantically irrelevant tokens. This dense distribution, while effective for broad context during pre-training, introduces noise that hinders fine-grained, spatially localized cues essential for dense, open-vocabulary prediction tasks. Researchers propose an inference-time modification, substituting the row-wise softmax in CLIP's final visual self-attention layers with the α-entmax transform. This transform applies a data-dependent threshold, effectively zeroing out contextually irrelevant dependencies and reallocating attention mass to the most pertinent tokens, thereby acting as an implicit denoiser. Evaluations on open-vocabulary tasks, including dense semantic segmentation (Pascal VOC, Pascal Context, ADE20K) and fine-grained retrieval (FG-OVD), demonstrate that the performance improvement from attention sparsification correlates with the degree to which baseline attention initially disperses off the target class.

Key takeaway

For Computer Vision Engineers optimizing CLIP models for dense open-vocabulary prediction, consider replacing the standard softmax self-attention with the α-entmax transform during inference. This modification can significantly reduce noise from irrelevant tokens, enhancing spatially localized cues crucial for tasks like semantic segmentation and fine-grained retrieval. You should evaluate the baseline attention's spread to predict potential performance gains from this sparsification technique.

Key insights

CLIP's dense attention creates noise; sparse α-entmax attention improves fine-grained open-vocabulary prediction.

Principles

• Softmax attention can dilute fine-grained signals.
• Sparsification can enhance relevant token focus.
• Performance gains link to baseline attention spread.

Method

Substitute row-wise softmax in CLIP's final visual self-attention layers with the α-entmax transform during inference. This transform applies a data-dependent threshold to zero out low scores.

In practice

• Apply α-entmax to CLIP for dense segmentation.
• Use sparsification for fine-grained retrieval tasks.
• Evaluate attention spread for potential gains.

Topics

• Sparse Attention
• CLIP
• Open-Vocabulary Prediction
• Semantic Segmentation
• Fine-Grained Retrieval
• α-entmax Transform

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language — SpotVMR efficiently trims query-relevant video clips for VMR, overcoming fixed-length downsampling issues and improving retrieval performance.
• Accelerating Text-to-Video Generation with Calibrated Sparse Attention — CalibAtt accelerates text-to-video generation by skipping consistently negligible attention computations via calibrated sparse attention.
• MiniMax M3 Just Killed Closed-Source Models — MiniMax M3 demonstrates sparse attention enables cost-effective, long-context, open-weights coding models.
• SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentation — SSP-SAM enhances SAM's language understanding for segmentation by integrating semantic and spatial cues via a novel prompt encoder.
• EntmaxKV: Support-Aware Decoding for Entmax Attention — EntmaxKV enables efficient, exact sparse decoding for long-context models by exploiting Entmax's zero-producing property.
• A Mechanism and Optimization Study on the Impact of Information Density on User-Generated Content Named Entity Recognition — Low information density in UGC is the root cause of NER model performance collapse, leading to "attention blunting."

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.