VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Models

2026-04-16 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

VisPCO is a new framework designed to optimize visual token pruning configurations in vision-language models (VLMs), addressing the quadratic computational growth from high-resolution image and video processing. Existing pruning methods often use predefined configurations without ensuring computational-performance optimality. VisPCO formulates this as a Pareto configuration optimization problem, employing continuous relaxation and straight-through estimators for gradient-based search, solved using the Augmented Lagrangian method. Experiments across 8 visual benchmarks show VisPCO accurately approximates the empirical Pareto frontier derived from grid search and demonstrates strong generalization across various pruning methods and VLM architectures. The framework also uses learnable kernel functions to analyze layer-wise pruning patterns, indicating that multi-step progressive pruning better captures VLMs' hierarchical compression structure, leading to improved accuracy-efficiency trade-offs over single-layer methods.

Key takeaway

For AI Engineers and Research Scientists working with vision-language models, VisPCO offers a systematic approach to optimize visual token pruning. By leveraging its Pareto configuration optimization, you can automatically identify configurations that achieve superior accuracy-efficiency trade-offs, moving beyond manual, predefined settings. This framework can significantly reduce computational costs for high-resolution inputs while maintaining performance, especially when adopting multi-step progressive pruning strategies.

Key insights

VisPCO optimizes visual token pruning in VLMs by formulating it as a Pareto configuration problem for optimal efficiency.

Principles

• Pruning configurations impact VLM computation-performance.
• Multi-step progressive pruning improves accuracy-efficiency.

Method

VisPCO uses continuous relaxation, straight-through estimators, and the Augmented Lagrangian method for gradient-based search to identify optimal pruning configurations.

In practice

• Apply VisPCO to find optimal VLM pruning settings.
• Consider multi-step progressive pruning for better trade-offs.

Topics

• Visual Token Pruning
• Vision-Language Models
• Pareto Optimization
• Budget-Aware Learning
• Gradient-Based Search

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Moving Beyond Diversity: Visual Token Pruning as Subspace Reconstruction for Efficient VLMs — SPARE prunes VLM visual tokens by minimizing reconstruction error and leveraging anti-relevance for context preservation.
• Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning — Progressive magnitude-based pruning enables effective neural network sparsification within a single training cycle.
• On Asymmetric Optimization of Reasoning and Perception in Vision-Language Model Post-Training — VLM post-training creates a perception-reasoning asymmetry due to token imbalance (SFT) or reward coupling (RL), hindering end-to-end performance.
• Understanding Pruning Regimes in Vision-Language Models Through Domain-Aware Layer Selection — Domain-aware pruning effectively reduces VLM depth by identifying and removing layers least critical to specific tasks.
• When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semantics — A two-stage visual token pruning method improves VLM efficiency by decoupling structural diversity from semantic relevance.
• SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation — SAFE-Pruner uses future-aware semantic attention to prune VLA model tokens, achieving efficient real-time robotic control without significant performance loss.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.