LensVLM: Selective Context Expansion for Compressed Visual Representation of Text

2026-07-07 · Source: Apple Machine Learning Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision, Natural Language Processing · Depth: Expert, quick

Summary

LensVLM, an inference framework and post-training recipe developed by Roy Xie et al. and published in July 2026, enhances Vision Language Models (VLMs) by enabling them to process text as rendered images while maintaining high accuracy under significant compression. VLMs typically struggle with accuracy as text images are compressed, making characters indistinguishable. LensVLM addresses this by initially scanning compressed images and then selectively expanding only relevant sections to their uncompressed form using learned tools. Built upon Qwen3.5-9B-Base, LensVLM achieves accuracy comparable to full-text processing at 4.3x effective compression and surpasses retrieval-based, text-, and visual-compression baselines by up to 10.1x effective compression across seven text QA benchmarks. The framework also extends its benefits to multimodal document and code understanding tasks, with performance gains increasing alongside compression levels. Analysis indicates that training improves visual compression robustness, and the model increasingly relies on expanded content as compression intensifies.

Key takeaway

For Machine Learning Engineers optimizing VLM deployment on resource-constrained devices, LensVLM offers a critical solution to the accuracy-compression trade-off. You can achieve up to 10.1x effective compression without significant accuracy loss by implementing its selective context expansion framework. Consider integrating this post-training recipe, especially for multimodal document and code understanding, to enhance efficiency while preserving performance.

Key insights

LensVLM selectively expands compressed visual text representations to maintain VLM accuracy at high compression ratios.

Principles

• Visual compression robustness improves with training.
• Models rely more on expanded content as compression increases.
• Tool-choice guidance optimizes expansion for content type.

Method

LensVLM scans compressed images, then uses learned tools to selectively expand relevant sections to their uncompressed resolution, integrating this into a VLM inference framework and post-training recipe.

In practice

• Apply text expansion for rendered text.
• Use high-resolution image expansion for native documents.
• Integrate with Qwen3.5-9B-Base for VLM tasks.

Topics

• Vision Language Models
• Text Compression
• Selective Context Expansion
• Qwen3.5-9B-Base
• Multimodal Document AI
• Code Understanding

Best for: Research Scientist, AI Engineer, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference — The framework selectively skips redundant visual computations at the operator level within MLLM Transformer layers, preserving accuracy while boosting efficiency.
• From Codebooks to VLMs: Evaluating Automated Visual Discourse Analysis for Climate Change on Social Media — VLMs can effectively analyze social media images for climate discourse, even with moderate per-image accuracy.
• Qwen3-VL: DeepStack Fusion, Interleaved-MRoPE, and a Native 256K Interleaved Context Window — Qwen3-VL integrates multimodal capabilities as a core design principle, enhancing both vision and language performance.
• Magnifying What Matters: Attention-Guided Adaptive Rendering for Visual Text Comprehension — VLMs localize text evidence without full utilization; adaptive rendering improves comprehension.
• Understanding Pruning Regimes in Vision-Language Models Through Domain-Aware Layer Selection — Domain-aware pruning effectively reduces VLM depth by identifying and removing layers least critical to specific tasks.
• KVCapsule: Efficient Sequential KV Cache Compression for Vision-Language Models with Asymmetric Redundancy — KVCapsule efficiently compresses VLM KV caches by leveraging asymmetric redundancy and dynamic attention patterns in vision tokens.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Apple Machine Learning Research.