Test-Time Training for Modality Order Consistency in Vision-Language Models

2026-07-23 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

Vision-language models (VLMs) exhibit a significant sensitivity to the input order of images and questions, with "image-first" prompting consistently outperforming "question-first" prompting by 6 to 26 percentage points across three models (InternVL2-8B, Qwen2.5-VL-7B, Qwen3-VL-8B) and three benchmarks (MMStar, RealWorldQA, AI2D). This "modality order failure" is a repeatable circuit-level issue, causally localized to a narrow mid-network region (around layers 18–20 in InternVL2-8B). Researchers developed an asymmetric test-time training (TTT) method that uses the stronger image-first prediction as a teacher signal to align the weaker question-first prediction. This TTT procedure substantially closes the question-first accuracy gap, with gains ranging from +5.2 to +26.1 percentage points, and surprisingly, also slightly improves image-first accuracy by +0.0 to +1.2 percentage points in most settings. The method involves instance-specific gradient updates on the language model backbone parameters within selected layer windows, with a moving teacher recomputed at each step.

Key takeaway

For AI Engineers deploying Vision-Language Models, you should explicitly test for modality order sensitivity, as image-first prompting consistently outperforms question-first. Implement asymmetric test-time training (TTT) to repair performance gaps in weaker orderings, potentially improving overall model consistency. This approach, targeting mid-network layers, can enhance robustness without requiring retraining, but be aware of increased inference costs due to per-instance gradient updates.

Key insights

Vision-language models exhibit a circuit-level sensitivity to modality order, which can be mitigated by asymmetric test-time training.

Principles

• Modality order impacts VLM performance.
• Asymmetric TTT can improve VLM robustness.
• Localized adaptation is more effective.

Method

An asymmetric test-time training (TTT) procedure uses the stronger image-first prediction as a detached teacher to align the weaker question-first prediction via KL divergence loss, applying instance-specific gradient updates to mid-network layers.

In practice

• Evaluate VLMs for modality order sensitivity.
• Apply asymmetric TTT to repair weaker prompt orderings.
• Target mid-network layers for efficient adaptation.

Topics

• Vision-Language Models
• Test-Time Training
• Modality Order Sensitivity
• Activation Patching
• Model Robustness
• Multimodal AI

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Test-Time Training for Modality Order Consistency in Vision-Language Models — Vision-language models exhibit a circuit-level failure due to input modality order, which test-time training can effectively mitigate.
• AIM: Asymmetric Information Masking for Visual Question Answering Continual Learning — Asymmetric Information Masking (AIM) prevents catastrophic forgetting in VQA by balancing VLM stability and plasticity.
• ISO-Bench: Benchmarking Multimodal Causal Reasoning in Visual–Language Models through Procedural Plans — Multimodal models struggle with causal reasoning, achieving only 0.57 F1 on ISO-Bench, far below human performance.
• Diagnosing Visual Ignorance in Vision-Language Models — Vision-Language Models frequently prioritize language priors over visual evidence due to internal routing failures and flawed benchmark evaluations.
• EASE-TTT: Evidence-Aligned Selective Test-Time Training for Long-Context Question Answering — Retrieved evidence can directly supervise query-side attention adaptation for improved long-context QA in smaller LLMs.
• Robustifying Vision-Language Models via Test-Time Prompt Adaptation — Adversarial attacks corrupt VLM representations, but semantic integrity can be preserved via distribution-level alignment.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.