Cosine Misleads: Auxiliary Losses Reshape Vision Language Models, Not Their Latents

2026-06-04 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study on latent visual reasoning (LVR) in vision-language models (VLMs) challenges the assumption that better alignment between supervised latent tokens and visual targets improves answer accuracy. Researchers tested five LVR variants and found a strong negative correlation (r=-0.94) between cosine alignment and accuracy. They introduced PRISM, a pair of inference-time diagnostics, revealing that supervised latents are largely bypassed, with corruption shifting accuracy by at most four points. The answer is decodable downstream of the latent, not at it, and this decodability gap predicts latent reliance. The findings suggest auxiliary objectives reshape the language model via shared parameters, rather than primarily through the nominally optimized latent variable.

Key takeaway

For AI Scientists and Research Scientists designing or evaluating vision-language models with latent visual reasoning, you should critically re-evaluate the common assumption that improved latent alignment directly enhances VLM accuracy. The evidence suggests auxiliary objectives primarily reshape the language model through shared parameters, bypassing the latents. Focus your efforts on understanding these broader architectural impacts rather than solely optimizing latent variables, and consider using diagnostics like PRISM to assess true latent utility.

Key insights

The core assumption that better latent alignment improves vision-language model accuracy is inverted.

Principles

• Cosine alignment negatively correlates with LVR accuracy.
• Supervised latents in LVR are often bypassed.
• Auxiliary losses reshape VLMs via shared parameters.

Method

PRISM diagnostics involve a linear probe to assess answer decodability and a corruption test to evaluate latent variable load-bearing capacity.

In practice

• Re-evaluate LVR latent alignment metrics.
• Test latent variable reliance with corruption.
• Analyze answer decodability downstream.

Topics

• Vision-Language Models
• Latent Visual Reasoning
• Auxiliary Losses
• Model Diagnostics
• Cosine Similarity
• Information Bottleneck

Best for: AI Scientist, Research Scientist

Related on AIssential

• The Geometry of Representational Failures in Vision Language Models — VLM multi-object failures stem from geometric interference between concept vectors in a shared latent space.
• On Asymmetric Optimization of Reasoning and Perception in Vision-Language Model Post-Training — VLM post-training creates a perception-reasoning asymmetry due to token imbalance (SFT) or reward coupling (RL), hindering end-to-end performance.
• Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning — Injecting fundamental geometric priors into VLMs directly enhances 3D spatial reasoning more effectively than VQA fine-tuning.
• Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs — Visual latents in MLLMs can be "silenced" by training objectives, but their reasoning capacity can be unleashed at inference time.
• Solving the “Whac-a-mole dilemma”: A smarter way to debias AI vision models — WRING debiases vision language models by rotating bias-related coordinates, avoiding the "Whac-A-Mole dilemma" of projection methods.
• Diagnosing Visual Ignorance in Vision-Language Models — Vision-Language Models frequently prioritize language priors over visual evidence due to internal routing failures and flawed benchmark evaluations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.