When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision, Natural Language Processing · Depth: Expert, extended

Summary

A unified framework analyzes attention sinks in Large Vision-Language Models (LVLMs), categorizing them into ViT-emerged sinks (V-sinks) from the vision encoder and LLM-emerged sinks (L-sinks) arising within deep LLM layers. The analysis, conducted on LLaVA-1.5-7B, reveals a trade-off: sinks effectively encode global scene-level priors but can suppress fine-grained visual evidence. V-sinks average 15-20 tokens per image, while L-sinks average 1-5 tokens per layer (L4-L18), both carrying rich, persistent scene information. To address this, Layer-wise Sink Gating (LSG) is proposed, a lightweight, plug-and-play module that dynamically scales V-sink attention contributions. LSG is trained for 2 epochs on 10K Cambrian-7M samples in approximately 2 hours on a single A100 GPU, achieving consistent performance gains of +1.56%p on MMStar and +3.08%p on CVBench benchmarks.

Key takeaway

For Machine Learning Engineers optimizing Large Vision-Language Models, recognize that attention sinks present a critical trade-off between global scene understanding and fine-grained local perception. You should consider implementing dynamic, layer-wise gating mechanisms, such as Layer-wise Sink Gating (LSG), to adaptively scale sink contributions. This approach, trainable with standard next-token prediction, can significantly improve performance on multimodal benchmarks, especially for vision-centric tasks requiring a balanced integration of global and local visual evidence.

Key insights

Attention sinks in LVLMs present a trade-off between global and local perception, requiring dynamic, layer-wise modulation.

Principles

• Visual sinks (V-sinks, L-sinks) encode global scene-level priors.
• Sink dominance can suppress fine-grained local perception.
• Optimal sink utilization is task- and layer-dependent.

Method

Layer-wise Sink Gating (LSG) employs a 2-layer MLP to predict layer-specific key scaling factors for V-sinks versus other visual tokens, conditioned on the final input token's hidden state.

In practice

• Categorize visual sinks into V-sinks and L-sinks for targeted analysis.
• Modulate attention contributions of sink tokens at specific LLM layers.
• Train lightweight gating modules with next-token prediction loss.

Topics

• Large Vision-Language Models
• Attention Sinks
• Vision Transformers
• Layer-wise Sink Gating
• Multimodal Benchmarks
• Attention Mechanisms

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective — Attention sinks on the first token in LLMs emerge early via a non-semantic P0 Sink Circuit.
• A Unifying View of Attention Sinks: Two Algorithms, Two Solutions — Visually similar attention sink patterns in transformers can hide two distinct underlying computational mechanisms.
• [Hands-On] GPT-OSS 바닥부터 구현하기 (3/5)- Self-Attention Mechanism — Implementing Self-Attention with GQA, Attention Sinks, and RoPE significantly optimizes Transformer model efficiency and contextual understanding.
• Understanding Pruning Regimes in Vision-Language Models Through Domain-Aware Layer Selection — Domain-aware pruning effectively reduces VLM depth by identifying and removing layers least critical to specific tasks.
• A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation — Attention sinks in Transformers arise from complex interactions, not single components, and can manifest through diverse circuits.
• You Only Index Once: Cross-Layer Sparse Attention with Shared Routing — Cross-layer sparse attention (CLSA) shares KV cache and routing indices across layers to improve LLM inference efficiency and quality.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.