VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

VISTA-Bench is a new benchmark designed to evaluate Vision-Language Models' (VLMs) understanding of visualized text embedded in images, contrasting it with pure-text queries. It comprises 1,500 samples across multimodal perception, reasoning, and knowledge, as well as unimodal understanding domains. Evaluations of over 20 representative VLMs, including Qwen3-VL-8B-Instruct and InternVL-3.5-8B, reveal a significant "modality gap": models performing well on pure-text queries degrade substantially when equivalent semantic content is presented as visualized text. This performance drop, exemplified by NEO-9B-SFT's 30.8-point decrease, is exacerbated by increased perceptual difficulty, such as small (9pt) or handwritten fonts, highlighting limited perceptual robustness as the primary bottleneck. The benchmark provides a principled framework to diagnose this limitation and guide the development of more unified language representations.

Key takeaway

For AI Scientists and Machine Learning Engineers developing or deploying VLMs, you must account for the "modality gap" identified by VISTA-Bench. Your models' performance on real-world inputs containing visualized text, especially with challenging renderings like small (e.g., 9pt) or stylized fonts, will likely be significantly lower than on pure-text queries. Prioritize training data that includes diverse text-as-pixels examples and consider robust OCR capabilities as a critical component to ensure reliable understanding across all input modalities.

Key insights

VLMs struggle significantly with text presented visually in images compared to pure text, revealing a critical "modality gap."

Principles

• Visualized text understanding degrades with perceptual difficulty.
• Stronger OCR ability correlates with a smaller modality gap.
• Multimodal context can attenuate the modality gap.

Method

VISTA-Bench construction involves data curation, a LaTeX-based rendering pipeline for diverse visual layouts (e.g., Arial 16pt, 800 pixels width), and VLM-based verification to ensure rendering fidelity.

In practice

• Test VLMs with diverse visualized text inputs.
• Prioritize VLM training on text-as-pixels data.
• Optimize rendering for legibility in VLM applications.

Topics

• Vision-Language Models
• Visualized Text Understanding
• Modality Gap
• Benchmark Evaluation
• Perceptual Robustness
• OCR Capabilities

Code references

• QingAnLiu/VISTA-Bench

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding — VLMs struggle with long, complex documents, showing length degradation and positional sensitivity.
• Multi-Agent Debate and Visual Information Extraction for SeePhys Pro: A 1st-Place Technical Report from ICML 2026 AI4Math Track 3 Challenge — The solution effectively bridges the visual-text modality gap in physics problems using a two-stage framework and multi-agent debate for superior accuracy.
• Disparities In Negation Understanding Across Languages In Vision-Language Models — VLMs show affirmation bias, with negation understanding varying significantly across diverse languages and models.
• From Pixels to Prompts: Vision-Language Models — Understanding Vision-Language Models requires a clear mental map beyond just knowing buzzwords.
• Magnifying What Matters: Attention-Guided Adaptive Rendering for Visual Text Comprehension — VLMs localize text evidence without full utilization; adaptive rendering improves comprehension.
• Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models — VLMs rewrite imperfect text based on language priors, unlike traditional OCR, impacting transcription faithfulness.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.