A robot operating system framework for using large language models in embodied AI

2026-03-16 · Source: Nature Machine Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, long

Summary

A new open-source robot operating system (ROS) framework, developed by Huawei Noah's Ark Lab and published in Nature Machine Intelligence (2026), enables autonomous robots to translate natural-language instructions into physical actions using large language models (LLMs). This framework connects an LLM agent to ROS, allowing it to convert LLM outputs into robot actions. It supports interchangeable execution modes, including inline code and behavior trees, and facilitates learning new atomic skills through imitation. The system continually refines these skills via automated optimization and reflection from human or environmental feedback. Extensive experiments demonstrate the framework's robustness, scalability, and versatility across diverse scenarios, such as long-horizon tasks, tabletop rearrangements, dynamic task optimization, and remote supervisory control, all utilizing open-source pretrained LLMs.

Key takeaway

For AI Scientists and Research Scientists developing embodied AI, this framework offers a robust method for integrating large language models with robot operating systems. You should explore this open-source implementation to enhance robot autonomy, particularly for complex, long-horizon tasks, and consider its approach to skill learning and refinement to improve system adaptability and performance.

Key insights

Connecting LLM agents to ROS enables versatile, robust, and scalable embodied AI through natural language.

Principles

• LLM outputs can be directly translated into robot actions.
• Skill learning benefits from imitation and iterative refinement.
• Open-source LLMs are viable for complex robotic tasks.

Method

The framework integrates an LLM agent with ROS, translating LLM outputs into robot actions. It supports inline code or behavior trees, learns skills via imitation, and refines them through optimization and feedback.

In practice

• Implement LLM-to-robot action translation for embodied AI.
• Utilize behavior trees for structured robot task execution.
• Incorporate human/environmental feedback for skill refinement.

Topics

• Large Language Models
• Embodied AI
• Robot Operating System
• Robot Control
• Skill Learning

Code references

• huawei-noah/HEBO

Best for: AI Scientist, Research Scientist, AI Researcher, Robotics Engineer, AI Engineer

Related on AIssential

• LLM-Powered Interactive Robotic Action Synthesis from Multimodal Speech, Gestures, and Music — LLMs can synthesize complex robot actions from multimodal human inputs like speech, gestures, and music.
• How robots learn: A brief, contemporary history — AI models and large datasets are revolutionizing robot learning, enabling adaptation and complex task execution.
• Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models — Embodied-R1.5 unifies diverse embodied reasoning capabilities into an 8B-parameter EFM, achieving SOTA performance and real-world generalization.
• Sergey Levine - Building LLMs for the Physical World - [Invest Like the Best, EP.465] — The core idea is building general robotic foundation models for universal control of embodied systems.
• LLM-Foraging: Large Language Models for Decentralized Swarm Robot Foraging — Integrating LLMs at key decision points in swarm robotics enhances adaptability and performance without costly re-optimization.
• Roundtables: Can AI Learn to Understand the World? — World models are emerging as a critical AI frontier to ground systems in real-world understanding, addressing LLM limitations in common sense and causality.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Nature Machine Intelligence.