Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning

2026-06-12 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Robotics & Autonomous Systems, Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

A novel Real2Sim2Real framework addresses challenges in tactile-only blind grasping for multi-fingered robotic hands, specifically the tactile sim-to-real gap and the limited expressiveness of sparse tactile signals. The approach integrates three key components: a Real2Sim tactile calibration pipeline that constructs a contact-calibrated digital-twin simulator, a layout-aware tactile encoder improving sparse observation expressiveness via self-supervised pretraining, and a tactile-conditioned Diffusion Policy aggregating successful grasp trajectories from object-specific reinforcement learning experts. Evaluated on a physical LEAP Hand equipped with 44 distributed tactile channels across 10 seen and 10 unseen objects, the deployed policy achieved a 27% real-world grasp success rate without real-world demonstrations or visual input. Simulation ablations confirmed that layout-aware pretraining enhances performance, and Real2Sim calibration increases tactile contact consistency between simulation and hardware.

Key takeaway

For Robotics Engineers developing dexterous manipulation systems, especially those facing sim-to-real transfer challenges with tactile feedback, this research offers a practical blueprint. You should prioritize implementing contact-event calibration to align simulated and real tactile signals, and integrate geometry-aware tactile representation learning. Adopting a diffusion-based policy can further improve generalization for blind grasping across diverse objects, even with sparse tactile inputs.

Key insights

Calibrated simulation and geometry-aware tactile representations enable blind dexterous grasping on physical robots.

Principles

• Contact-event calibration reduces sim-to-real discrepancy.
• Spatial layout priors enhance sparse tactile observations.
• Diffusion policies aggregate diverse grasping behaviors.

Method

The framework involves Real2Sim tactile calibration, self-supervised pretraining of a layout-aware tactile encoder, and aggregating object-specific RL expert trajectories into a tactile-conditioned Diffusion Policy.

In practice

• Use binary tactile signals for robust sim-to-real transfer.
• Pretrain tactile encoders with privileged geometric supervision.
• Employ diffusion policies for multimodal action generation.

Topics

• Blind Grasping
• Dexterous Manipulation
• Tactile Sim-to-Real
• Diffusion Policy
• Tactile Representation Learning
• LEAP Hand

Code references

• NVIDIAGameWorks/kaolin

Best for: Research Scientist, AI Scientist, Robotics Engineer, Machine Learning Engineer

Related on AIssential

• Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning — Combining calibrated simulation, geometry-aware tactile encoding, and diffusion policies enables blind dexterous grasping.
• Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation — Center-of-Pressure (CoP) enables robust sim-to-real dexterous manipulation by preserving dense, physics-grounded tactile information.
• Dense Force Estimation with an Event-based Optical Tactile Sensor — Event-based tactile sensors can reconstruct dense 3D force fields by mapping displacement data to forces via iFEM and neural networks.
• TAM: Torque Adaptation Module for Robust Motion Transfer in Manipulation — TAM enables robust robot policy transfer by adapting torques using proprioceptive history, trained purely in simulation.
• Mana: Dexterous Manipulation of Articulated Tools — Mana reinterprets dexterous articulated tool manipulation as an animation problem, enabling zero-shot sim-to-real transfer via a coarse-to-fine pipeline.
• SECOND-Grasp: Semantic Contact-guided Dexterous Grasping — Integrating semantic reasoning with physical stability enhances robotic dexterous grasping performance and adaptability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.