Reward Design for Physical Reasoning in Vision-Language Models

2026-04-15 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

A systematic reward ablation study investigates how reward design influences Vision Language Model (VLM) physical reasoning. Researchers compared four reward signals—format compliance, answer accuracy, a composite rubric, and a novel internal attention-weight reward—for Group Relative Policy Optimization (GRPO)-based VLM training. The study utilized IBM Granite Vision 3.3 (2B) and evaluated performance on PhyX, a 3,000-problem benchmark covering six physics domains and six reasoning types in multiple-choice and open-ended formats. Results indicate that GRPO with accuracy-based rewards generally outperforms Supervised Fine-Tuning (SFT) across most domains. Reward design does not uniformly improve performance but induces domain-specific reasoning behaviors. Accuracy-based rewards yielded the strongest overall gains, while rubric rewards improved structured reasoning quality without consistent accuracy improvements. The attention-based reward enhanced spatial reasoning, increasing spatial relation accuracy from 0.27 to 0.50, despite degrading performance in symbolic domains.

Key takeaway

For research scientists developing Vision Language Models, understanding reward design is crucial for improving physical reasoning. Your choice of reward signal directly impacts domain-specific performance; accuracy-based rewards offer the most robust general improvements, while attention-based rewards are particularly effective for spatial reasoning tasks. You should experiment with different reward types to optimize VLM performance for specific physics domains and reasoning challenges.

Key insights

Reward design significantly shapes VLM physical reasoning, with accuracy-based rewards offering the strongest overall gains.

Principles

• Reward design induces domain-specific reasoning.
• Accuracy-based rewards provide strong overall gains.
• Internal attention-weight rewards enhance spatial reasoning.

Method

A systematic reward ablation study for GRPO-based VLM training compared four reward signals: format compliance, answer accuracy, a composite rubric, and an internal attention-weight reward.

In practice

• Prioritize accuracy-based rewards for general VLM physical reasoning.
• Consider rubric rewards for structured reasoning quality.
• Explore attention-weight rewards for spatial reasoning tasks.

Topics

• Vision Language Models
• Physical Reasoning
• Reward Design
• Group Relative Policy Optimization
• PhyX Benchmark

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Reward Design for Physical Reasoning in Vision-Language Models — Reward design significantly shapes VLM physical reasoning, with accuracy and attention-based signals driving domain-specific performance gains.
• Prioritizing the Best: Incentivizing Reliable Multimodal Reasoning by Rewarding Beyond Answer Correctness — Reasoning-answer inconsistency in multimodal RL can be mitigated by rewarding the quality of reasoning, not just answer correctness.
• SVoT: State-aware Visualization-of-Thought for Spatial Reasoning via Reinforcement Learning — SVoT improves MLLM spatial reasoning by generating verifiable intermediate states and visualizations via reinforcement learning.
• STRIDE: Strategic Trajectory Reasoning via Discriminative Estimation for Verifiable Reinforcement Learning — STRIDE enhances RLVR by using verifiable outcomes to identify and reward outcome-discriminative strategic patterns, improving credit assignment.
• A Pre-Registered Causal Partition of Self-Consistency Elicitation and Reward Design in RLVR — The Δ_naive estimand in RLVR conflates self-consistency elicitation with genuine reward design, leading to biased attribution.
• Vero: An Open RL Recipe for General Visual Reasoning — Broad data diversity and task-routed rewards are crucial for general visual reasoning in open VLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.