A Picture Costs a Thousand Words — Literally

2026-07-26 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Intermediate, medium

Summary

Vision Transformers (ViTs) consume a high number of tokens for images, significantly increasing costs in multimodal AI models. ViTs process images by dividing them into 16x16 pixel patches, treating each as a token. The self-attention mechanism then incurs a quadratic cost (N²) based on the token count (N), meaning a 1024x1024 image with 16-pixel patches generates 4,096 tokens and roughly 16.8 million token pairs for attention. Major AI providers—OpenAI, Anthropic, and Google's Gemini—employ distinct billing models for image tokens, resulting in up to a 2.7x cost difference for an identical 4032x3024 phone photo. Research, exemplified by LLaVA-PruMerge, reveals that up to 95% of visual tokens often carry minimal information. This has spurred development of token reduction strategies, including larger patches, dynamic merging (Qwen2-VL), on-the-fly pruning (FastV, PruMerge), fixed-budget encoders (BLIP-2, Flamingo), and windowed attention (Swin Transformer), all aimed at making image processing more efficient.

Key takeaway

For AI Engineers and ML Architects optimizing multimodal model costs, understanding visual token consumption is critical. Your image processing strategy directly impacts billing, with a 2.7x cost spread possible for identical inputs across providers. Downscale and crop images before uploading, and avoid re-sending them in multi-turn chats. Consider OCR for documents to significantly reduce token expenses. This proactive management of visual input can substantially lower operational costs and improve model efficiency.

Key insights

Vision Transformers' quadratic attention cost for image tokens drives high expense, despite most visual tokens being redundant.

Principles

• Visual token cost scales quadratically with token count.
• Image resolution quadratically increases token count.
• Most visual tokens carry minimal marginal information.

Method

Researchers reduce visual token costs via larger patches, dynamic merging, on-the-fly pruning, fixed-budget encoders like Q-Former, or windowed attention to break the N² wall.

In practice

• Downscale images before uploading to models.
• Crop images to relevant regions for cost and accuracy.
• Extract text via OCR for documents instead of pixels.

Topics

• Vision Transformers
• Multimodal AI
• Tokenization Costs
• Visual Token Reduction
• Attention Mechanisms
• Image Processing

Best for: AI Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• Pixels for Programs? A Cross-Provider Case Study of Input-Token Accounting for Source Code as Text and Images — Sending source code as compact images to LLM APIs can significantly reduce reported input tokens, but with provider-specific break-even points.
• Tokens Are the New Bytes — Tokens are becoming the architectural unit for AI, defining how language models process meaning, similar to bytes in traditional computing.
• AI costs how much? GitHub Copilot users react to new usage-based pricing system. — GitHub Copilot's shift to usage-based billing reveals significant AI inference costs previously absorbed by providers.
• Claude Opus 4.7 is AMAZING! Full Breakdown + Testing Results — Claude Opus 4.7 enhances image processing and design, but at a significantly higher operational cost.
• Your AI Bill Has Two Dials. You Are Probably Only Turning One. — Optimizing "tokens per task" through compression, RAG, caching, and routing yields multiplicative AI cost savings.
• AdaTok: Self-Budgeting Image Tokenization with Quality-Preserving Dynamic Tokens — Self-budgeting image tokenization dynamically adjusts token counts based on visual complexity, improving efficiency and fidelity.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.