Japan Science and Technology Agency Develops NVIDIA-Powered Moonshot Robot for Elderly Care

2026-01-08 · Source: NVIDIA Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Advanced, quick

Summary

NVIDIA is developing a lightweight, real-time AI model that enables robots to predict 3D force distributions from a single RGB image, mimicking human intuition for object interaction. The model is trained exclusively on simulation data, where image data is paired with corresponding contact forces, which are then converted into smooth and stable distributions. Domain randomization is applied to ensure the model generalizes effectively to real-world images. This predictive capability allows robots to compute optimal directions for manipulating target objects, such as lifting a yellow box while minimizing disturbance to surrounding brown objects, a task successfully demonstrated with a real robot. Further simulation confirmed significant reduction in disturbance across various overlapping object scenes.

Key takeaway

For robotics engineers developing manipulation systems, this approach offers a method to equip robots with a human-like understanding of interaction forces. You can leverage simulation-trained models to predict 3D force distributions from standard RGB camera input, enabling more precise and less disruptive object handling in cluttered environments. Consider integrating domain randomization during training to ensure robust performance in real-world scenarios.

Key insights

Robots can predict 3D interaction forces from RGB images using simulation-trained AI.

Principles

• Simulation data can generalize to real-world robotics.
• Domain randomization enhances model robustness.

Method

Pair image data with contact forces in simulation, convert forces to stable distributions, apply domain randomization, then train a model to predict 3D force maps from RGB images.

In practice

• Use predicted force maps for robot manipulation.
• Minimize disturbance to surrounding objects.

Topics

• Robotics
• Force Prediction
• Simulation-to-Real Transfer
• Object Manipulation
• Domain Randomization

Best for: AI Scientist, Research Scientist, AI Engineer, Robotics Engineer, AI Researcher

Related on AIssential

• Mana: Dexterous Manipulation of Articulated Tools — Mana reinterprets dexterous articulated tool manipulation as an animation problem, enabling zero-shot sim-to-real transfer via a coarse-to-fine pipeline.
• How robots learn: A brief, contemporary history — AI models and large datasets are revolutionizing robot learning, enabling adaptation and complex task execution.
• Training a Robotic Arm Using MuJoCo and JAX on AMD Hardware with ROCm™ — Reinforcement learning with MuJoCo and JAX on AMD ROCm enables robust robotic arm training for complex manipulation tasks.
• Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation — Joint velocity action space significantly improves sim-to-real performance for vision-based robotic picking and pushing tasks.
• NVIDIA launches Newton 1.0 physics engine for industrial robot training — NVIDIA has released Newton 1.0, a production-ready open-source physics simulation engine designed for training robots in contact-rich manipulation and locomotion tasks, built on NVIDIA Warp and OpenUS…
• From Seeing to Simulating: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluation — WorldComposer generates diverse, high-fidelity simulation environments from real-world panoramas for robust robot learning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by NVIDIA Blog.