Develop Physical AI Reasoning, World, and Action Models with NVIDIA Cosmos 3

2026-06-01 · Source: NVIDIA Technical Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Cloud Computing & IT Infrastructure · Depth: Expert, long

Summary

NVIDIA has released Cosmos 3, a frontier foundation model for physical AI that unifies physical reasoning, world generation, and action generation within a single open model. This release includes open-source Cosmos 3 models, training scripts, deployment tools, and six synthetic data generation datasets for applications like robotics and autonomous driving. The architecture features a Mixture-of-Transformers with a 16B-parameter Cosmos 3 Nano for workstation-grade compute (NVIDIA RTX PRO 6000 GPU) and a 64B-parameter Cosmos 3 Super for datacenter deployment (NVIDIA Hopper/Blackwell GPUs). Cosmos 3 supports various input/output modalities and leads on benchmarks such as VANTAGE-Bench and PAI-Bench. NVIDIA also introduced the Cosmos Human Evaluation (HUE) framework for objective video generation quality assessment. Training recipes for Supervised Fine-Tuning and action post-training are provided, alongside deployment options via NVIDIA NIM microservices, which offer optimizations like NVFP4 quantization and vLLM for efficient inference.

Key takeaway

For Machine Learning Engineers developing physical AI systems, NVIDIA Cosmos 3 offers a unified, open-source foundation model to streamline development. You can utilize its 16B-parameter Nano model for efficient edge inference or the 64B-parameter Super model for high-quality datacenter workloads. Leverage the provided training recipes for Supervised Fine-Tuning with your custom data and deploy optimized models using NVIDIA NIM microservices to accelerate your projects.

Key insights

Cosmos 3 unifies physical AI reasoning, world generation, and action generation into a single, open foundation model.

Principles

• Unify reasoning and generation for simplified physical AI development.
• Open-source models and datasets foster reproducibility.
• Objective human evaluation improves model quality assessment.

Method

Cosmos 3 employs a Mixture-of-Transformers with a VLM-based Reasoner tower for interpretation and a diffusion-based Generator tower for physics-aware video and action sequence output.

In practice

• Use Cosmos 3 Nano (16B) for real-time robotics on NVIDIA RTX PRO 6000.
• Deploy Cosmos 3 via NIM microservices for optimized inference.
• Adapt Cosmos 3 with SFT using custom video or action-labeled data.

Topics

• Physical AI
• NVIDIA Cosmos 3
• Foundation Models
• Robotics
• Autonomous Vehicles
• Synthetic Data Generation
• NVIDIA NIM Microservices

Code references

• nvidia/Cosmos
• DAGroup-PKU/ReVidgen
• SHI-Labs/physical-ai-bench
• NVIDIA/cosmos-framework

Best for: AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Welcome NVIDIA Cosmos 3: The First Open Omni-model for Physical AI Reasoning and Action — NVIDIA Cosmos 3 unifies physical AI capabilities into a single omni-model for comprehensive world understanding and action generation.
• Scale Synthetic Data and Physical AI Reasoning with NVIDIA Cosmos World Foundation Models — NVIDIA Cosmos WFMs accelerate physical AI development through advanced synthetic data generation and multimodal reasoning.
• Nvidia Intros Data Factory, Robotics Models in Physical AI Push — Nvidia's new tools and models aim to overcome physical AI data collection challenges through synthetic data generation and advanced world models.
• The Breakthrough Moment for Physical AI, Powered by NVIDIA Cosmos — NVIDIA Cosmos is a foundation model that uses synthetic data and unified representations to advance physical AI.
• Nvidia Launches Physical AI Models for Robots — Nvidia's new physical AI models and frameworks aim to democratize intelligent robot development and deployment.
• #347 Let's Get Physical with AI with Ivan Poupyrev, CEO at Archetype AI — Physical AI leverages foundation models to transform real-world sensor data into actionable insights, recommendations, and automation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by NVIDIA Technical Blog.