Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models

2026-07-27 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision, Natural Language Processing · Depth: Expert, extended

Summary

Vision Language Models (VLMs) often rewrite imperfect text into plausible forms rather than transcribing faithfully, a behavior traditional OCR systems avoid. A new multilingual perturbation benchmark, FaithC4, comprising 1,455 single-page documents in English, Chinese, and Korean, was introduced to evaluate this. Evaluating 15 systems, researchers found general-purpose VLMs degrade by up to 4.5 points WER on English, OCR-specialized VLMs by 0.2–2 points, and traditional OCR by less than 0.6 points. Probing Qwen3-VL-4B revealed rewriting occurs when a perturbed word's final layer FFN representation stays close to its original encoding. Short words (4–6 characters) are rewritten up to 10% of the time, with rewriting dropping to 0% above 8 characters. Perturbing 5% of words also increased errors on unperturbed text by 2–5x.

Key takeaway

For AI Scientists and Machine Learning Engineers deploying VLMs for document processing, recognize that general-purpose models prioritize plausibility over literal transcription, especially with imperfect text. This rewriting behavior, particularly for short words, can introduce significant errors and propagate to unperturbed text. You should consider traditional OCR or carefully selected OCR-specialized VLMs for applications requiring high transcription faithfulness, such as legal or medical documents, to avoid silent data corruption.

Key insights

VLMs rewrite imperfect text based on language priors, unlike traditional OCR, impacting transcription faithfulness.

Principles

• VLM rewriting correlates with language prior strength.
• Rewriting is gated by internal representation similarity.
• Short words are more prone to VLM rewriting.

Method

The FaithC4 benchmark uses C4 text, applies scramble, random, or visual perturbations to 8% of eligible words, then renders them as PDFs for VLM evaluation.

In practice

• Use traditional OCR for literal transcription needs.
• Select OCR-specialized VLMs for moderate robustness.
• Consider word length when assessing VLM transcription risk.

Topics

• Vision Language Models
• Optical Character Recognition
• Transcription Faithfulness
• FaithC4 Benchmark
• Text Perturbation
• Document Understanding

Code references

• mindee/doctr

Best for: Research Scientist, AI Engineer, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs — RL-finetuned VLMs face an accuracy-faithfulness trade-off, compromising robustness and CoT consistency.
• Short-form Text Rewriting with Phi Silica — Targeted adaptation of Small Language Models (SLMs) can significantly improve their performance for precision-critical short-form text rewriting tasks.
• VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text? — VLMs struggle significantly with text presented visually in images compared to pure text, revealing a critical "modality gap."
• SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding — VLMs struggle with long, complex documents, showing length degradation and positional sensitivity.
• Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation — VLMs exhibit substantial script-dependent bias, failing to consistently process identical content across different orthographies.
• Beyond Supervised Clarification: Input Rewriting with LLMs for Dialogue Discourse Parsing — Unsupervised input rewriting for dialogue discourse parsing often causes regressions, highlighting the need for rewritability prediction.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.