Beyond Attention Scores: SVD-Based Vision Token Pruning for Efficient Vision-Language Models

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, long

Summary

SVD-Prune is a novel, training-free, and plug-and-play token pruning method for Vision-Language Models (VLMs) that enhances computational efficiency by addressing the high demands of processing long sequences of vision tokens. Unlike existing methods relying on local heuristics like attention scores, which suffer from positional bias, SVD-Prune uses Singular Value Decomposition (SVD) to identify and preserve vision tokens contributing most to the dominant global variance. It decomposes the vision token feature matrix and selects top-K tokens using statistical leverage scores. Experiments on LLaVA-1.5-7B, with 336x336 input images (initially 576 tokens), show SVD-Prune consistently outperforms prior methods. It achieves 53.77 on GQA at 64 tokens and maintains stable performance at 32 and 16 tokens, with drops of 8.38 and 8.86 points on GQA, and 3.39 and 4.17 points on TextVQA. This method reduces total FLOPs by up to 84.8%, from 3.45 T to 0.52 T FLOPs when reducing from 576 to 16 tokens.

Key takeaway

For Machine Learning Engineers deploying Vision-Language Models on resource-constrained edge devices, SVD-Prune offers a significant efficiency gain. You can achieve up to 84.8% FLOPs reduction by pruning vision tokens to 16 or 32, while maintaining robust performance on tasks like GQA and TextVQA. This training-free, plug-and-play method avoids extensive retraining, making it ideal for adapting VLMs without compromising critical reasoning capabilities. Consider integrating SVD-Prune to optimize inference costs and memory footprint.

Key insights

SVD-Prune uses global variance and leverage scores for training-free VLM token pruning, outperforming attention-based methods.

Principles

• Vision tokens contribute unevenly to VLM reasoning.
• Attention scores are biased for token importance.
• Global variance captures essential visual structure.

Method

SVD-Prune performs SVD on the vision token feature matrix, truncates to dominant variance, estimates token contribution via leverage scores, then selects and prunes tokens.

In practice

• Apply SVD-Prune to LLaVA-1.5-7B for efficiency.
• Prune vision tokens to 16 or 32 for extreme compression.
• Use a plug-and-play, training-free pruning approach.

Topics

• Vision-Language Models
• Token Pruning
• Singular Value Decomposition
• Computational Efficiency
• LLaVA-1.5-7B
• Edge AI

Code references

• anoncvlab/svdPrune

Best for: AI Engineer, Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• Moving Beyond Diversity: Visual Token Pruning as Subspace Reconstruction for Efficient VLMs — SPARE prunes VLM visual tokens by minimizing reconstruction error and incorporating anti-relevance for context preservation.
• VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Models — VisPCO optimizes visual token pruning in VLMs by formulating it as a Pareto configuration problem for optimal efficiency.
• Moving Beyond Diversity: Visual Token Pruning as Subspace Reconstruction for Efficient VLMs — SPARE prunes VLM visual tokens by minimizing reconstruction error and leveraging anti-relevance for context preservation.
• Dispatch-Aware Ragged Attention for Pruned Vision Transformers — Dispatch overhead, not compute, bottlenecks pruned ViT attention at short sequence lengths.
• When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semantics — A two-stage visual token pruning method improves VLM efficiency by decoupling structural diversity from semantic relevance.
• Understanding Pruning Regimes in Vision-Language Models Through Domain-Aware Layer Selection — Domain-aware pruning effectively reduces VLM depth by identifying and removing layers least critical to specific tasks.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.