Thinking in Pictures: A Diagnostic Study of Visual vs. Textual Chain-of-Thought Reasoning in Vision-Language Models

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

The paper "Thinking in Pictures: A Diagnostic Study of Visual vs. Textual Chain-of-Thought Reasoning in Vision-Language Models" by Ben Jenkins, presented at ALVR 2026, investigates when Vision-Language Models (VLMs) should use visual versus textual Chain-of-Thought (CoT) reasoning. It introduces VisCoT-Diag, a diagnostic benchmark of 1,200 instances across five visual reasoning categories, used to compare four CoT paradigms across four VLMs. The study found a significant "modality gap," where textual CoT degraded performance by up to 17.5% on spatial transformation and 13.2% on multi-object tracking. Conversely, visual CoT improved performance by up to 23.1%. The research identified three failure modes: spatial state collapse, transformation hallucination, and tracking loss. An adaptive modality routing approach achieved 73.1% accuracy, outperforming V-CoT-everywhere at 68.9%.

Key takeaway

For Machine Learning Engineers developing Vision-Language Models, you should critically evaluate the modality of your CoT reasoning. Employing visual CoT is crucial for tasks involving spatial transformation and object tracking, where it can yield performance gains of up to 23.1%. Conversely, use textual CoT for compositional counting. Implementing adaptive modality routing can further boost accuracy to 73.1%, avoiding common failure modes like spatial state collapse.

Key insights

Visual Chain-of-Thought significantly outperforms textual CoT for inherently visual tasks in Vision-Language Models.

Principles

• Textual CoT degrades VLM performance on visual tasks.
• Visual CoT improves VLM performance on visual tasks.
• Adaptive modality routing enhances VLM accuracy.

Method

VisCoT-Diag, a 1,200-instance benchmark across five visual reasoning categories, was used to compare four CoT paradigms across four VLMs, identifying failure modes and evaluating adaptive routing.

In practice

• Use visual CoT for VLM spatial reasoning tasks.
• Employ textual CoT for VLM compositional counting.
• Implement adaptive modality routing for VLMs.

Topics

• Vision-Language Models
• Chain-of-Thought (CoT) Reasoning
• Visual Reasoning Benchmarks
• Modality Gap
• Spatial Transformation
• Object Tracking

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Chain-of-Thought Degrades Visual Spatial Reasoning Capabilities of Multimodal LLMs — Chain-of-Thought prompting degrades visual spatial reasoning in multimodal LLMs due to shortcut learning and hallucination.
• VLADriveBench: Evaluating CoT-Action Relationship in VLA for Autonomous Driving — VLADriveBench evaluates the causal link between VLA model chain-of-thought and autonomous driving actions, revealing discrepancies in CoT utility.
• Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models — VLMs exhibit "answer inertia" and inconsistent modality reliance, even with Chain-of-Thought reasoning.
• Thinking with Visual Grounding — Vision-language models enhance reasoning by explicitly grounding natural-language thoughts to specific image regions.
• CaVe-VLM-CoT: An Interpretable Vision-Language Model Framework — CaVe-VLM-CoT enforces evidence-grounded reasoning in VLMs through a closed-loop, reflection-based RAG agent to combat hallucinations.
• Beyond Visual Cues: CoT-Enhanced Reasoning for Semi-supervised Medical Image Segmentation — Integrating Chain-of-Thought reasoning with LLM-generated linguistic knowledge improves semi-supervised medical image segmentation by resolving visual-semantic mismatches.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.