VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression

2026-07-14 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

VisCo is a training-efficient self-compression framework designed to address the substantial inference latency and memory overhead in vision-language models (VLMs) caused by processing numerous visual tokens. Unlike existing training-free methods that degrade performance under high compression or training-based approaches that incur retraining costs and compromise VLM priors, VisCo reuses the pretrained VLM itself as an intrinsic compressor. This parameter-sharing autoencoder compresses visual information into a small set of memory tokens and transfers hierarchical information during decoding. Experimental results demonstrate that VisCo outperforms previous methods across all evaluated compression ratios, showing greater improvements under aggressive compression, and maintains stability even in an extreme single-token configuration. Furthermore, combining the learned memory tokens with original visual tokens can enhance the base model, indicating VisCo captures complementary representations beyond mere compression.

Key takeaway

For Machine Learning Engineers optimizing vision-language model inference, VisCo offers a compelling solution to reduce latency and memory overhead. If you are struggling with high visual token counts, consider implementing this self-compression framework. It allows you to reuse existing VLM backbones for efficient compression without costly retraining, potentially improving your base model's performance by capturing complementary representations. This approach provides superior compression across various ratios, even in extreme single-token scenarios.

Key insights

Pretrained vision-language models can intrinsically compress visual tokens, enhancing efficiency and potentially improving base model performance.

Principles

• Strong information encoding is key for visual token compression.
• Pretrained VLMs possess underutilized intrinsic encoding capabilities.
• Self-compression via parameter-sharing autoencoder is efficient.

Method

VisCo operates as a parameter-sharing autoencoder. It compresses visual information into a small set of memory tokens and transfers hierarchical information from the encoding stage to the decoding stage.

In practice

• Achieve superior visual token compression across all ratios.
• Maintain stability even with extreme single-token compression.
• Enhance base VLM performance using learned memory tokens.

Topics

• Visual Token Compression
• Vision-Language Models
• Large Language Models
• Autoencoders
• Inference Optimization
• Parameter Sharing

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning — OccamToken uses register-anchored relative evidence testing for training-free, budget-adaptive VLM token pruning, improving efficiency.
• Progressive Cramming: Reliable Token Compression and What It Reveals — Perfect token reconstruction through cramming does not guarantee semantic utility or preserved downstream model capabilities.
• Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality — MACCO improves VLM compositional understanding by cross-modal masked concept reconstruction and auxiliary alignment objectives.
• KVCapsule: Efficient Sequential KV Cache Compression for Vision-Language Models with Asymmetric Redundancy — KVCapsule efficiently compresses VLM KV caches by leveraging asymmetric redundancy and dynamic attention patterns in vision tokens.
• Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models — SPpruner reduces VLM computational cost by mimicking human "Focus-then-Context" perception to retain salient subjects and their structural context.
• PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understanding — PARCEL dynamically partitions visual feature extraction using anchored resampling to overcome LVLM compression limitations.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.