[P] I open-sourced a synth framework for creating physics-simulated humanoids in Unity with MuJoCo -- train them with on-device RL and interact in VR

2026-03-04 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Emerging Technologies & Innovation · Depth: Advanced, quick

Summary

A new open-source system, comprising three Unity packages, enables the creation and training of physics-based humanoid characters for mixed reality on Meta Quest. The `synth-core` package converts Daz Genesis 8 or Mixamo characters into physics-simulated humanoids with MuJoCo rigid-body dynamics, mesh-based collision, and configurable joints. `synth-training` facilitates on-device Soft Actor-Critic (SAC) reinforcement learning using TorchSharp, supporting Mac, Windows, and Quest without an external Python server, and includes features like prioritized experience replay and motion reference tooling. The `synth-vr` package integrates these characters into mixed reality environments on Meta Quest, allowing physical interaction via hand tracking, passthrough rendering with depth occlusion, and ambient light estimation. The entire system is built on Unity 6, MuJoCo, TorchSharp, and Meta XR SDK, and is released under the Apache-2.0 license.

Key takeaway

For AI Scientists and Research Scientists developing embodied agents, this open-source system offers a robust foundation for creating and training physics-based humanoids directly within Unity and deploying them to mixed reality on Quest. You can leverage its on-device reinforcement learning capabilities and MuJoCo integration to accelerate development of interactive virtual beings, bypassing the need for external Python training servers and simplifying deployment to mobile VR platforms.

Key insights

This system enables physics-based humanoid creation and on-device reinforcement learning for mixed reality.

Principles

• On-device RL training is feasible for complex agents.
• Mixed reality interaction benefits from physics simulation.

Method

Import a humanoid model into Unity, create a Synth prefab, add a ContinuousLearningSkill component, and build for Quest to enable physics simulation and on-device reinforcement learning.

In practice

• Convert Daz Genesis 8 characters to physics models.
• Train SAC agents directly on Quest hardware.
• Integrate physics humanoids into MRUK environments.

Topics

• Physics-based Humanoids
• Reinforcement Learning
• Mixed Reality
• Unity Engine
• MuJoCo Physics

Best for: AI Scientist, Research Scientist, AI Engineer, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Building a Custom Drones MuJoCo Environment [P] — A new MuJoCo-based GitHub package provides multi-agent RL drone environments for community use and feedback.
• PhysDrift: Bridging the Embodiment Gap in Humanoid Co-Speech Motion Generation — Directly generating robot-native co-speech motions from speech overcomes the embodiment gap, improving humanoid expressiveness.
• Nvidia releases DreamDojo, a robot ‘world model’ trained on 44,000 hours of human video — DreamDojo uses vast human video data to teach robots physical world interaction, significantly cutting training time and cost.
• Utilizing AMD Schola and UnrealRoboticsLab with AMD ROCm™ Software to Train a Robotic Arm — Integrating Unreal Engine, MuJoCo via URLab, and Schola enables high-fidelity, physics-accurate robotic arm reinforcement learning on AMD hardware.
• Disney Characters Coming to Life at NVIDIA GTC 2026 — Disney uses physical AI, Newton, and Kamino to create expressive, autonomous robotic characters.
• Why Do Humanoid Robots Still Struggle With the Small Stuff? — Despite AI advancements, humanoid robots still struggle with fundamental physics like force control for reliable real-world interaction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.