Test-Time Training for Modality Order Consistency in Vision-Language Models

2026-07-22 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Vision-Language Models · Depth: Expert, quick

Summary

Vision-language models exhibit sensitivity to the presentation order of images and questions, a semantically irrelevant change. Across three distinct models and three benchmarks, image-first prompting consistently outperformed question-first prompting, revealing a repeatable modality order failure. Researchers designed an order-consistent test-time training method that substantially closes this modality-order gap across all evaluated settings. This method also surprisingly yields consistent performance improvements in the stronger image-first branch compared to the baseline, effectively bootstrapping both orderings towards mutual consistency. Activation patching localized the ordering failure to a narrow mid-network region where representations sharply diverge between prompt orders, a misalignment that the test-time training method successfully repairs across layers. These findings identify modality-order sensitivity as a circuit-level failure in VLMs, demonstrating that simple, asymmetric test-time adaptation can effectively mitigate it and enhance baseline performance.

Key takeaway

For Machine Learning Engineers deploying vision-language models, you should rigorously test your models for modality order sensitivity. If your VLM exhibits performance degradation with question-first inputs, consider implementing asymmetric test-time training. This approach can not only close the performance gap between input orderings but also consistently improve your model's overall accuracy beyond its baseline. Your deployment strategy should account for potential input order variations.

Key insights

Vision-language models exhibit a circuit-level failure due to input modality order, which test-time training can effectively mitigate.

Principles

• Modality order significantly influences VLM performance.
• Test-time training can enhance VLM robustness and consistency.
• Activation patching helps localize internal model failures.

Method

An order-consistent test-time training method adapts VLMs by repairing mid-network representation misalignment that arises from different input modality orders.

In practice

• Test VLMs with both image-first and question-first inputs.
• Apply test-time training to improve VLM consistency.
• Investigate mid-network activations for VLM failure diagnosis.

Topics

• Vision-Language Models
• Test-Time Training
• Modality Order Sensitivity
• Activation Patching
• Model Robustness
• Neural Network Analysis

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Test-Time Training for Modality Order Consistency in Vision-Language Models — Vision-language models exhibit a circuit-level sensitivity to modality order, which can be mitigated by asymmetric test-time training.
• Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models — Multimodal LLMs exhibit significant order sensitivity, with current prompt-level mitigations proving insufficient for general robustness.
• Semantic Robustness Certification for Vision-Language Models — This framework certifies Vision-Language Model robustness against semantic variations using text prompts as proxies.
• Listening makes Vision Clear for VLMs — VLM vision-language consistency evaluation requires prompt-side semantics to avoid decoding drift and structural token bias.
• AIM: Asymmetric Information Masking for Visual Question Answering Continual Learning — Asymmetric Information Masking (AIM) prevents catastrophic forgetting in VQA by balancing VLM stability and plasticity.
• Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models — VLMs exhibit "answer inertia" and inconsistent modality reliance, even with Chain-of-Thought reasoning.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.