RCT: A Robot-Collected Touch-Vision-Language Dataset for Tactile Generalization

2026-06-30 · Source: Artificial Intelligence · Field: Technology & Digital — Robotics & Autonomous Systems, Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

RCT (Robotic Contact Tactile) is a new robot-collected touch-vision-language dataset designed to address tactile generalization challenges for robots manipulating open-world objects. Comprising 29,279 tactile frames, RCT captures full robot presses on 122 industrial reference materials across 7 categories, utilizing three DIGIT sensors at various contact positions. Each press is preserved as a contact sequence, enabling rigorous held-out evaluation across materials, categories, and sensors. The dataset reveals that frame-random splits can introduce near-duplicate observations, and removing contact-sequence overlap significantly reduces tactile-to-text Recall@1 by 17.7 percentage points. Furthermore, holding out materials during training causes a sharp performance drop, with held-out-material Recall@1 averaging 25.1 +/- 6.1%. RCT facilitates reproducible contact-sequence-aware evaluation and underscores novel-material generalization as a critical hurdle for robotic tactile perception.

Key takeaway

For Machine Learning Engineers developing tactile perception models, you must rigorously evaluate generalization to novel materials. Frame-random data splits can inflate performance; instead, ensure your test sets contain entirely held-out contact sequences and materials. This approach, facilitated by datasets like RCT, will reveal true model capabilities and highlight the critical challenge of novel-material generalization, guiding more robust model development for open-world robotic applications.

Key insights

Robot tactile perception struggles with novel material generalization, exacerbated by data collection biases.

Principles

• Data splits must avoid contact-sequence overlap.
• Held-out material evaluation is crucial.
• Uniform sampling improves contrastive training.

Method

The proposed method involves collecting full robot presses on diverse materials with multiple sensors, preserving contact sequences, and using uniform press sampling for contrastive training.

In practice

• Evaluate tactile models on unseen materials.
• Implement contact-sequence-aware data splits.
• Use uniform press sampling for training.

Topics

• Robotic Tactile Perception
• Dataset Generalization
• DIGIT Sensors
• Contact Sequences
• Material Recognition
• Machine Learning Datasets

Best for: Research Scientist, AI Scientist, Robotics Engineer, Machine Learning Engineer

Related on AIssential

• TACTFUL: Tactile-Driven Exploration For Object Localization and Identification in Confined Environments — Tactile sensing, structured learning, and real-hardware training enable robots to identify objects without vision.
• UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generation — UniTac is the first unified multimodal model to integrate tactile understanding and generation across diverse sensors using a dual-level representation.
• Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning — Calibrated simulation and geometry-aware tactile representations enable blind dexterous grasping on physical robots.
• Would you say capture-time semantic annotation for robot trajectories is a solved problem? [R] — Raw robot teleoperation data lacks critical semantic context unrecoverable post-hoc, creating a bottleneck for contact-rich tasks.
• CLASP: Language-Driven Robot Skill Selection and Composition using Task-Parameterized Learning — CLASP integrates TP-KMPs with VLMs for data-efficient, language-driven robot skill learning, selection, and composition.
• DAIMON Robotics Wants to Give Robot Hands a Sense of Touch — Tactile sensing is crucial for advanced robot manipulation, enabling the VTLA model to overcome VLA limitations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.