HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds

2026-04-15 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

HY-World 2.0 is a multi-modal world model framework that generates and reconstructs 3D world representations from various inputs, including text prompts, single-view images, multi-view images, and videos. Building upon HY-World 1.0, this iteration synthesizes high-fidelity, navigable 3D Gaussian Splatting (3DGS) scenes using a four-stage method: Panorama Generation with HY-Pano 2.0, Trajectory Planning with WorldNav, World Expansion with WorldStereo 2.0, and World Composition with WorldMirror 2.0. Key innovations include enhanced panorama fidelity, improved 3D scene understanding, an upgraded WorldStereo with consistent memory, and a refined WorldMirror for universal 3D prediction. The framework also introduces WorldLens, a high-performance 3DGS rendering platform with features like automatic IBL lighting and efficient collision detection. Experiments show HY-World 2.0 achieves state-of-the-art performance among open-source models, comparable to the closed-source Marble model.

Key takeaway

For research scientists developing 3D world models, HY-World 2.0 offers a robust open-source framework with state-of-the-art performance. You should explore its modular architecture and released weights to advance your own research in multi-modal 3D scene generation and reconstruction, potentially leveraging its innovations in panorama fidelity and consistent memory for view generation.

Key insights

HY-World 2.0 is a multi-modal framework for generating and reconstructing high-fidelity 3D worlds from diverse inputs.

Principles

• Multi-modal input enhances 3D world generation.
• Modular design supports iterative model improvement.
• Consistent memory improves keyframe-based view generation.

Method

HY-World 2.0 employs a four-stage method for 3D world generation: Panorama Generation (HY-Pano 2.0), Trajectory Planning (WorldNav), World Expansion (WorldStereo 2.0), and World Composition (WorldMirror 2.0).

In practice

• Generate 3D scenes from text prompts.
• Reconstruct 3D worlds from multi-view images.
• Explore 3D worlds with character support.

Topics

• Multi-Modal World Models
• 3D Gaussian Splatting
• 3D World Generation
• 3D World Reconstruction
• Interactive 3D Rendering

Best for: Research Scientist, AI Scientist, Computer Vision Engineer, Machine Learning Engineer

Related on AIssential

• World Tracing: Generative Pixel-Aligned Geometry Beyond the Visible — World Tracing unifies pixel-aligned depth and generative completion for comprehensive 3D geometry reconstruction.
• WorldKV: Efficient World Memory with World Retrieval and Compression — WorldKV uses retrieval and compression to maintain persistent world memory in video diffusion models efficiently.
• MoVerse: Real-Time Video World Modeling with Panoramic Gaussian Scaffold — MoVerse creates interactive 3D worlds from single images by separating world construction from observation rendering using panoramic diffusion and Gaussian scaffolds.
• Global-Local Monte Carlo Tree Search in Vision-Language Models for Text-to-3D Indoor Scene Generation — PRM-guided MCTS with hierarchical scene representation enables LVLMs to generate realistic 3D indoor scenes by correcting errors and balancing search.
• Rein3D: Reinforced 3D Indoor Scene Generation with Panoramic Video Diffusion Models — Rein3D integrates 3D Gaussian Splatting with panoramic video diffusion to restore and refine 3D indoor scenes from sparse inputs.
• Open Thread 421 — Genie 3 creates persistent, explorable AI-generated worlds by leveraging extensive spatial data and advanced world memory.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.