Engine-Native Editable 3D World Reconstruction with Objects and Lighting

2026-07-23 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

Lumera (Light-aware Unified Engine-native Reconstruction and Assembly) introduces a new benchmark and pipeline for engine-native, light-aware 3D scene parsing from a single image, addressing limitations of existing methods that lack editable object instances and lights for standard engines. The Lumera-2K dataset, derived from 2,513 UE5 projects, comprises 3.73M components, 63M object instances, 102.6K engine-native parametric lights, and 95.1K camera views. The pipeline utilizes Lumera-Box and Lumera-Light, which adapt VLM to parse object boxes and parametric light tuples (x,y,z,r,g,b,I). Lumera-Box achieved superior detection, geometry, semantic, and layout scores (merged mAP 0.1141, IoU-B 0.2472, F-score 0.2762) against competitors like DetAny3D, though WildDet3D showed stronger anchor recall. Lumera-Light recovered almost all non-empty scenes (recall 0.998) but had limited individual-light localization (F1 0.209 at 0.5 m), with matched lights showing a median position error of 0.261 m. This work establishes parametric lights as a measurable target for editable scenes.

Key takeaway

For Computer Vision Engineers developing 3D scene reconstruction systems, Lumera demonstrates a viable path for generating engine-native, editable assets. You should consider integrating parametric light parsing and object box detection using VLM-adapted approaches to achieve higher fidelity and editability. This work highlights the importance of robust light localization and relation structure for future advancements in creating truly interactive 3D environments from single images.

Key insights

Lumera enables engine-native, editable 3D world reconstruction from a single image, including parametric objects and lights.

Principles

• Parametric lights are a measurable editable-scene target.
• VLM adapts for parsing object boxes and light tuples.

Method

The Lumera pipeline adapts VLM for object box and parametric light parsing, integrates per-object mesh reconstruction, HDR environment estimation, and uses a bounded agentic refinement loop for assembly.

In practice

• Use Lumera-2K for 3D scene reconstruction research.
• Evaluate light localization using parametric light metrics.
• Adapt VLM for parsing specific scene components.

Topics

• 3D Scene Reconstruction
• Engine-Native Assets
• Parametric Lights
• Object Detection
• VLM (Vision-Language Models)
• Unreal Engine 5

Best for: Research Scientist, AI Scientist, Computer Vision Engineer

Related on AIssential

• Engine-Native Editable 3D World Reconstruction with Objects and Lighting — Lumera enables engine-native 3D scene reconstruction from a single image, parsing objects and parametric lights for editable environments.
• Video Scene Graph Generation Using VLMs — VLMs enable open-vocabulary video scene graph generation, overcoming fixed ontology limitations for richer video understanding.
• Do Image Editing Models Understand Lighting? — Generative image editing models demonstrate remarkable consistency with real-world lighting physics but still have room for improvement.
• Vision-Reasoning-Guided Occlusion Removal from Light Fields — Combining light field integration with vision-language models robustly removes occlusions for enhanced scene recovery.
• 3D-Aware VLMs with Implicit and Explicit Geometries — VLM-IE3D improves vision-language models' 3D spatial reasoning by integrating implicit and explicit geometries from RGB video.
• 🍓Surface Light Tokenizer🍓 👉Apple unveils LITO a novel latent flow matching model enables... — LITO uses latent flow matching to convert images into high-quality 3D models via compact surface light field representations.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.