Better Reward Models For Robots

2026-07-03 · Source: The Batch | DeepLearning.AI | AI News & Insights - www.deeplearning.ai · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, short

Summary

Researchers at Stanford University and UC Berkeley, including Tony Lee, Andrew Wagenmaker, and Karl Pertsch, developed RoboReward, a family of 4 billion-parameter and 8 billion-parameter vision-language reward models. These models enhance reinforcement learning for robots by providing more effective rewards than previous general-purpose models, narrowing the performance gap with labor-intensive handcrafted functions. A key innovation involves augmenting training data with negative examples, created by relabeling positive actions or truncating successful videos, to teach models the distinction between success and failure. RoboReward 8B achieved a 0.665 mean absolute error on the RoboRewardBench test dataset, outperforming models like GPT-5 (0.811) and Gemini Robotics-ER 1.5 (0.906). In real-world tests, a robot trained with RoboReward 8B achieved 50 percent success for picking and 80 percent for opening a drawer, significantly better than Gemini Robotics-ER 1.5.

Key takeaway

For Machine Learning Engineers developing robot control policies, RoboReward offers a robust alternative to labor-intensive handcrafted reward functions. You should integrate the 4 billion-parameter or 8 billion-parameter RoboReward models into your reinforcement learning pipelines to achieve more effective training across diverse robot types and tasks. This approach significantly reduces the need for task-specific engineering, allowing you to focus on policy optimization rather than reward function design.

Key insights

Augmenting robot action datasets with negative examples significantly improves vision-language reward models' effectiveness.

Principles

• Negative examples are crucial for reward model learning.
• Relabeling positive data creates effective negative examples.
• Truncating successful videos generates partial progress examples.

Method

The authors built a diverse robot-action dataset, standardized task descriptions, augmented data with negative examples using GPT-5 mini and Qwen3-4B-Instruct-2507, and assigned progress scores. They then fine-tuned Qwen3-VL models to predict these scores as rewards.

In practice

• Use RoboReward models for robot RL training.
• Generate negative data by relabeling or truncating.
• Evaluate reward models with RoboRewardBench.

Topics

• Reinforcement Learning
• Robot Control
• Vision-Language Models
• Reward Models
• Data Augmentation
• RoboReward

Best for: Research Scientist, AI Scientist, Robotics Engineer, Machine Learning Engineer

Related on AIssential

• Progress Reward Modeling for Robotic Learning: A Comprehensive Survey — Progress rewards are crucial for robotic learning, offering dense, intermediate feedback to overcome limitations of terminal success signals in complex tasks.
• Jump-Start Reinforcement Learning with Vision-Language-Action Regularization — VLAJS improves RL efficiency in robotics by integrating sparse, annealed VLA guidance with PPO for better exploration.
• Reward Learning through Ranking Mean Squared Error — R4 uses a novel ranking MSE loss to learn RL rewards from ordinal human ratings, outperforming prior methods with less feedback.
• Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation — Noisy autonomous vision-language evaluation, with correction, provides scalable reward signals for RL in GUI agents.
• Reward Hacking in Reinforcement Learning — Reward hacking exploits reward function flaws, leading to unintended behaviors in RL agents, especially LLMs.
• What is wrong with reinforcement learning? (Ep. 82) — Reinforcement learning, despite its successes, faces significant limitations in real-world applicability due to inherent complexities.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The Batch | DeepLearning.AI | AI News & Insights - www.deeplearning.ai.