RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Software Development & Engineering · Depth: Expert, extended

Summary

RoboDojo introduces a unified sim-and-real benchmark for comprehensively evaluating generalist robot manipulation policies. It features 42 simulation tasks across five capability dimensions—Generalization, Memory, Precision, Long-Horizon, and Open—and 18 real-world tasks spanning three collaborative bimanual robot embodiments (ARX X5, Piper, Piper X). The platform integrates heterogeneous parallel simulation in Isaac Sim, achieving 77.4 interactions/s, and provides RoboDojo-RealEval for reproducible real-world testing, completing 180 physical trials in approximately 3.4 hours. XPolicyLab unifies policy development and deployment, integrating 30 policies. Evaluation reveals a substantial performance gap, with the best policy achieving only an 8.80% average simulation success rate and 12.8% real-world success, far below human experts, highlighting critical limitations in precision, memory, and open-semantic manipulation.

Key takeaway

For AI Scientists and Machine Learning Engineers developing generalist robot manipulation policies, you should prioritize addressing the identified gaps in precision, memory, and open-semantic understanding. Leverage RoboDojo's unified sim-and-real benchmark to systematically diagnose policy limitations, iterate efficiently in simulation, and validate robustness under standardized real-world conditions. Focus on developing policies with smoother low-level action priors, contact-aware feedback, and robust semantic-to-action grounding to bridge the significant performance gap to human experts.

Key insights

RoboDojo unifies sim-and-real evaluation to diagnose generalist robot manipulation policies' capabilities and limitations.

Principles

• Generalist policies lack balanced capability development.
• Precision tasks expose major bottlenecks in control.
• Open-semantic manipulation remains largely unsolved.

Method

RoboDojo uses 42 simulation tasks (5 dimensions) and 18 real-world tasks (3 embodiments). It leverages Isaac Sim for heterogeneous parallel simulation and RoboDojo-RealEval for standardized, reproducible physical testing, integrated via XPolicyLab.

In practice

• Integrate policies via XPolicyLab for unified evaluation.
• Utilize heterogeneous parallel simulation for rapid iteration.
• Submit to the public leaderboard for verified policy comparison.

Topics

• Robot Manipulation
• Generalist Policies
• Sim-to-Real Transfer
• Benchmarking
• Isaac Sim
• XPolicyLab
• Embodied AI

Code references

• stephane-caron/pink
• isaac-sim/IsaacSim
• thu-ml/RDT2

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation — Robot manipulation benchmarks must incorporate observed human behavior to evaluate complex reasoning beyond single-image instructions.
• From Seeing to Simulating: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluation — WorldComposer generates diverse, high-fidelity simulation environments from real-world panoramas for robust robot learning.
• ATOM-Bench: A Real-World Benchmark for Atomic Skills and Compositional Generalization in Manipulation Policies — Current robotic manipulation policies struggle with fine motor skills and compositional generalization, even when atomic skills are strong.
• RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies — RoboLab offers a high-fidelity simulation benchmark to assess robotic policy generalization and sensitivity to external factors.
• Why Robots Still Struggle With Simple Tasks (And What Might Finally Change That) | Karol Hausman, Co-Founder & CEO of Physical Intelligence — Combining internet-scale LLM knowledge with robot motion data enables general physical intelligence for complex tasks.
• $μ$VLA: On Recurrent Memory for Partially Observable Manipulation in VLA Models — Isolating minimal recurrence in VLA models significantly enhances performance in partially observable robotic manipulation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.