GraspIT: A Dataset Bridging the Sim-to-Real gap and back for Validated Grasping SE(3) Pose Generation

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Robotics & Autonomous Systems, Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, short

Summary

GraspIT is a new dataset designed to bridge the simulation-to-real-world gap for robust robotic grasping. It provides photorealistic RGB-D observations, physically validated grasp quality annotations, and a principled connection between simulated and real environments. The dataset was created by annotating tabletop scenes in NVIDIA Isaac Sim using a four-stage physical slip-test on parallel Franka Panda instances. This process generated trajectory-reachability checks and continuous quality scores from approximately 2.3 million grasp candidates, with 83% passing as "good" (s≥0.50) and 17% serving as graded hard negatives. A Real↔Sim loop then back-projects these labels onto 100 real-world scenes. The release includes about 316,000 annotated RGBD frame sets across 1035 simulated and 100 real scenes, complete with instance masks, 6-DoF poses, physical object properties, and scored 6-DoF grasps. All associated tools are open-source and Docker-containerized, supporting high-resolution demonstrations for manipulation policy learning.

Key takeaway

For Robotics Engineers developing robust grasping systems, GraspIT offers a critical resource to overcome sim-to-real challenges. You can leverage its physically validated 6-DoF grasp data and hard negatives to train more reliable models, reducing real-world deployment failures. Utilize the open-source tools and Docker containers to integrate this dataset efficiently into your development pipeline for manipulation policy learning.

Key insights

GraspIT bridges sim-to-real for robotic grasping with physically validated, high-quality 6-DoF grasp pose data.

Principles

• Physical slip-tests validate grasp quality.
• Real↔Sim loop enhances label transfer.
• Hard negatives improve model robustness.

Method

GraspIT's method involves annotating Isaac Sim scenes with a four-stage physical slip-test using Franka Panda robots, generating quality scores and trajectory checks, then back-projecting labels to real scenes.

In practice

• Train robust grasping models.
• Develop tabletop manipulation policies.
• Benchmark sim-to-real transfer.

Topics

• Robotic Grasping
• Sim-to-Real Transfer
• GraspIT Dataset
• NVIDIA Isaac Sim
• 6-DoF Pose Estimation
• Manipulation Policy Learning

Best for: Research Scientist, AI Scientist, Robotics Engineer, Machine Learning Engineer

Related on AIssential

• Human Universal Grasping — Human egocentric grasping data and flow-matching models enable robots to achieve universal, zero-shot object manipulation.
• EquiDexFlow: Contact-Grounded SE(3)-Equivariant Dexterous Grasp Generative Flows — Jointly predicting grasp kinematics and contact forces ensures physically stable dexterous grasps.
• A Cross-view Fusion Framework for Robust 6-DoF Grasp Pose Estimation — A cross-view fusion framework uses contrastive learning and cylindrical integration to robustly estimate 6-DoF grasp poses, especially in occluded views.
• RCT: A Robot-Collected Touch-Vision-Language Dataset for Tactile Generalization — Robot tactile perception struggles with novel material generalization, exacerbated by data collection biases.
• Grasp-Then-Plan with Failure Attribution: A Closed Two-Stage Framework for Precise and Generalizable Robotic Manipulation — GTP-FA improves robotic manipulation by attributing failures to optimize distinct grasping and planning stages.
• Mana: Dexterous Manipulation of Articulated Tools — Mana reinterprets dexterous articulated tool manipulation as an animation problem, enabling zero-shot sim-to-real transfer via a coarse-to-fine pipeline.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.