Tactile and Vision Conditioned Contact-Centric Control for Whole-Arm Manipulation

2026-07-10 · Source: Artificial Intelligence · Field: Technology & Digital — Robotics & Autonomous Systems, Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

TACTIC (Tactile and Vision Conditioned Contact-Centric Control) is a novel receding-horizon controller designed for whole-arm manipulation, a challenging robotics domain where robots make direct, distributed contact with the environment. This system addresses issues like tightly coupled motion and contact forces, occluded contact states, and physical inconsistencies in purely learned rollouts. TACTIC integrates RGB-D, distributed tactile sensing, and a compact 2D proximity representation. It couples a learned, action-conditioned latent dynamics model with analytical kinematics through contact Jacobians, enabling predictive rollouts of future contact configurations and interaction forces. These rollouts feed into a sampling-based Model Predictive Control (MPC) planner, which uses contact Jacobian-based projections for action sampling and objectives over predicted proximity and forces to balance task progress with whole-arm force regulation. TACTIC consistently outperforms state-of-the-art model-based and model-free methods in simulation and demonstrates real-world efficacy on a robot performing multi-contact tasks such as repositioning a manikin and navigating a 3D dynamic maze.

Key takeaway

For Robotics Engineers developing control systems for complex physical interaction tasks, TACTIC demonstrates a robust approach to whole-arm manipulation. Its hybrid predictive model, combining learned dynamics with analytical kinematics and multi-modal sensing, significantly improves physical consistency and performance in multi-contact scenarios. You should consider integrating similar contact-centric hybrid models and sampling-based MPC with contact-aware action sampling to enhance robot control in environments requiring distributed contact and force regulation.

Key insights

TACTIC integrates vision, tactile sensing, and analytical kinematics for robust, contact-centric whole-arm manipulation control.

Principles

• Whole-arm manipulation requires explicit contact modeling.
• Hybrid predictive models improve physical consistency.
• Contact Jacobians steer force-modulating actions.

Method

TACTIC uses a contact-centric hybrid predictive model combining RGB-D, tactile sensing, and 2D proximity. It couples learned latent dynamics with analytical kinematics via contact Jacobians for rollouts, then integrates these into a sampling-based MPC planner with contact-aware action sampling.

In practice

• Apply TACTIC for multi-contact robot tasks.
• Use distributed tactile sensing for complex interactions.

Topics

• Whole-Arm Manipulation
• Tactile Sensing
• Model Predictive Control
• Hybrid Robotics Models
• Contact Dynamics
• Robotics Control

Best for: Research Scientist, AI Scientist, Robotics Engineer

Related on AIssential

• Inference-time Policy Steering via Vision and Touch — ViTaL integrates vision and touch for robust robot policy steering in contact-rich manipulation.
• Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation — Center-of-Pressure (CoP) enables robust sim-to-real dexterous manipulation by preserving dense, physics-grounded tactile information.
• WristMimic: Full-Body Humanoid Control with Wrist-Guided Manipulation — The wrist serves as a critical, contact-free bridge for decoupling whole-body kinematic control from contact-rich, object-driven finger manipulation.
• Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning — Calibrated simulation and geometry-aware tactile representations enable blind dexterous grasping on physical robots.
• Exo2EgoPose: Leveraging Exocentric Demonstrations for Vision-Language guided Egocentric 3D Hand Pose Forecasting — Exocentric demonstrations can effectively guide egocentric 3D hand pose forecasting, bridging human-robot action gaps.
• Interaction Dynamics Modeling and Predictive Control for Safe Steerable Catheter--Tissue Interaction — Predictive control with explicit force constraints enables safe, precise steerable catheter-tissue interaction.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.