CARE: Competence-Aware Reward Shaping for Adaptive Reasoning Length in Video-MLLMs

2026-06-18 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

CARE is a novel competence-aware reward shaping framework designed to optimize adaptive reasoning length in multimodal video reasoning, particularly for Video-MLLMs. It addresses the limitations of traditional reinforcement learning methods that employ inflexible reasoning-length control, which can hinder early exploration or encourage redundant reasoning in competent models. CARE operates by maintaining a smoothed competence estimate using an exponential moving average of pass rates, progressively shifting reward preferences from exploration-oriented long-form reasoning to efficiency-oriented concise reasoning. The framework also normalizes reasoning effort with batch-level statistics and incorporates a posterior amplifier to boost rewards for strong performance on challenging samples. Integrated into the GRPO training pipeline without inference-time overhead, CARE consistently improves reasoning accuracy, stabilizes reinforcement learning, and significantly enhances token efficiency across various video reasoning and general video understanding benchmarks. It demonstrates an inverted-U trajectory of reasoning length during training, resulting in shorter, more informative reasoning traces at convergence.

Key takeaway

For Machine Learning Engineers developing Video-MLLMs, if you are struggling with inefficient reasoning or unstable reinforcement learning, consider implementing competence-aware reward shaping. CARE's approach of dynamically adjusting reasoning length based on model proficiency can significantly enhance token efficiency and stabilize training. You should explore integrating this framework into your GRPO-based pipelines to achieve shorter, more informative reasoning traces without incurring additional inference-time overhead.

Key insights

CARE adaptively optimizes reasoning length in Video-MLLMs by shaping rewards based on the model's evolving competence.

Principles

• Reward shaping should adapt to model competence.
• Distinguish reasoning verbosity from task complexity.
• Progressive training stages can guide exploration to efficiency.

Method

CARE estimates competence via exponential moving average, routes training into progressive stages, normalizes reasoning effort with batch statistics, and uses a posterior amplifier for strong performance.

In practice

• Integrate competence-aware reward shaping into GRPO pipelines.
• Monitor reasoning length for an inverted-U trajectory.
• Utilize provided source code for Video-MLLM development.

Topics

• Video-MLLMs
• Reinforcement Learning
• Reward Shaping
• Adaptive Reasoning
• Token Efficiency
• Multimodal Reasoning

Code references

• 1Pansy/Video-CARE

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Reasoning as Intersection: Consensus-Frame Alignment for Visual Focus in Video-MLLMs — Consensus Frame GRPO (CF-GRPO) improves Video-MLLM reasoning by aligning visual evidence with a consensus frame prior, using process-level rewards without temporal annotations.
• CARE-RL: Capability-Aware Reinforcement Learning for Mitigating Cross-Domain Conflicts — CARE-RL combines protocol-aware reward generation and capability-aware optimization to resolve multi-domain RL conflicts.
• sGPO: Trading Inference FLOPs for Training Efficiency in RLVR — sGPO uses cheap inference to profile query difficulty, optimizing RLVR training by reducing wasted FLOPs.
• Reward Design for Physical Reasoning in Vision-Language Models — Reward design significantly shapes VLM physical reasoning, with accuracy and attention-based signals driving domain-specific performance gains.
• Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization — GNDPO stabilizes on-policy distillation for MLLM reasoning by normalizing KL scores into batch-level relative advantages.
• CODA: Difficulty-Aware Compute Allocation for Adaptive Reasoning — Adaptive reasoning optimizes compute by aligning reasoning depth with task difficulty, avoiding overthinking simple problems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.