AnchorPrune: Relevance-Anchored Contextual Expansion for Visual Token Pruning

2026-07-08 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

AnchorPrune is a training-free framework designed to reduce substantial inference costs in large vision-language models (VLMs) caused by thousands of redundant visual tokens from high-resolution inputs. It addresses conflicts in existing pruning methods by first constructing a protected relevance anchor, then expanding it with complementary visual context. AnchorPrune adaptively determines the anchor size based on the novelty profile of relevance-ranked tokens and allocates the remaining budget using importance-weighted novelty. This ordered design prevents indispensable query cues from being displaced while improving visual coverage. Lightweight and architecture-aware, AnchorPrune requires no retraining or model modification. It consistently improves the accuracy-efficiency trade-off over baselines, particularly under severe compression, preserving 97.6% of LLaVA-NeXT-7B's full-token performance using only 160 of 2,880 visual tokens.

Key takeaway

For Machine Learning Engineers deploying large vision-language models with high-resolution inputs, you should integrate AnchorPrune to significantly reduce inference costs. This training-free framework preserves 97.6% of LLaVA-NeXT-7B's full-token performance using only 160 visual tokens, offering a superior accuracy-efficiency trade-off, especially under aggressive compression. Implement this lightweight solution to deploy high-resolution VLM applications more efficiently without requiring model retraining or modification.

Key insights

Relevance-anchored contextual expansion effectively prunes visual tokens for efficient multimodal inference in large vision-language models.

Principles

• Prioritize relevance anchors before contextual expansion.
• Determine anchor size adaptively from token novelty profiles.
• Allocate remaining budget via importance-weighted novelty.

Method

AnchorPrune constructs a protected relevance anchor, then expands it with complementary visual context. It adaptively sizes the anchor from the novelty profile of relevance-ranked tokens and allocates the remaining budget through importance-weighted novelty.

In practice

• Apply AnchorPrune to LLaVA-NeXT-7B to achieve 97.6% performance with 160 visual tokens.
• Integrate AnchorPrune for improved accuracy-efficiency in image and video VLMs.

Topics

• AnchorPrune
• Visual Token Pruning
• Vision-Language Models
• Multimodal Inference
• Inference Cost Optimization
• LLaVA-NeXT-7B

Code references

• MULTI-cau/AnchorPrune

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A — Visual token pruning in MLLMs is best approached as task-conditioned evidence search, not static ranking.
• DCP-Prune: Ultra-Low Token Pruning with Distribution Consistency Preservation — DCP-Prune stabilizes ultra-low token pruning by preserving feature distribution consistency, preventing performance degradation in vision models.
• Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models — SPpruner reduces VLM computational cost by mimicking human "Focus-then-Context" perception to retain salient subjects and their structural context.
• Doing More With Less: Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling — Unstructured pruning can enhance LLM test-time scaling performance, challenging prior assumptions about pruning's detrimental effects.
• Moving Beyond Diversity: Visual Token Pruning as Subspace Reconstruction for Efficient VLMs — SPARE prunes VLM visual tokens by minimizing reconstruction error and leveraging anti-relevance for context preservation.
• Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs — Sink tokens, semantically uninformative but high-attention visual tokens, hinder fine-grained video understanding in Video LLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.