Visual-Advantage On-Policy Distillation for Vision-Language Models

2025-10-27 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

Visual-Advantage On-Policy Distillation (VA-OPD) is a novel method addressing the challenge of strengthening visual reliance in vision-language models (VLMs) during on-policy knowledge distillation. Standard distillation often improves output quality but fails to make students depend on fine-grained visual input. Researchers introduce "visual advantage" (VA), a token-level log-probability difference indicating teacher reliance on visual detail. VA is sparse, with the top 10% of tokens carrying ~93% of visual supervision. VA-OPD leverages this by reweighting rollouts based on trajectory-averaged VA and grouping tokens into high-VA and low-VA categories for separate KL averaging. Evaluated on Qwen3-VL models (4B, 8B, 32B teachers to 2B student) across Geometry3K and ViRL39K datasets, VA-OPD consistently outperforms standard on-policy distillation on eight benchmarks, showing gains up to +3.8 on Math Avg and +2.5 on Visual Avg, with improvements scaling with teacher size and data.

Key takeaway

For AI Scientists and Machine Learning Engineers optimizing VLM distillation, you should adopt Visual-Advantage On-Policy Distillation (VA-OPD) to genuinely enhance your student models' reliance on fine-grained visual details. This method, by focusing distillation on vision-critical tokens identified via "visual advantage," prevents the dilution of crucial visual signals. Implementing VA-OPD can yield significant accuracy improvements, up to +3.8 Math Avg and +2.5 Visual Avg, especially on vision-intensive tasks, ensuring your models learn to "read the image" rather than just mimic text.

Key insights

Visual supervision in VLM on-policy distillation is sparse, concentrated in few tokens, and diluted by uniform KL averaging.

Principles

• Visual advantage (VA) quantifies token-level visual dependency.
• High-VA tokens carry the primary visual supervision signal.
• Dilution of visual signal hinders VLM visual reliance.

Method

VA-OPD reweights student rollouts by trajectory-averaged VA and applies token-level KL averaging separately within high-VA and low-VA groups.

In practice

• Use VA to identify vision-critical tokens in VLM outputs.
• Prioritize distillation signal on high-VA tokens.
• Degrade images (e.g., 10% resolution) to compute VA.

Topics

• Vision-Language Models
• Knowledge Distillation
• On-Policy Learning
• Visual Advantage
• Qwen3-VL
• Mathematical Reasoning

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation — Visually grounded cues, recoverable by the student, enhance multimodal reasoning distillation more effectively than answer-side privilege.
• World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning — World2VLM distills world model imagination into VLMs to improve dynamic spatial reasoning without inference-time overhead.
• The Hidden Evolution of Disguised Visual Context inside the VLM — Visual token integration paradigms in VLMs dictate internal representation evolution, affecting feature utilization and task performance.
• Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs — VIGIL mitigates MLLM visual laziness and hallucinations by causally grounding responses to visual input, penalizing "blind confidence" via counterfactual alignment.
• What does RL improve for Visual Reasoning? A Frankenstein-Style Analysis — RL primarily refines mid-to-late VLM layers, enhancing vision-to-reasoning alignment, not uniform visual perception.
• Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation — Self-distillation from regional crops to full images improves MLLM fine-grained visual understanding.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.