GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers

2026-04-22 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

GeoRelight is a novel Multi-Modal Diffusion Transformer designed to jointly solve the ill-posed task of relighting a person from a single 2D photo and reconstructing their 3D geometry. Traditional methods often struggle with error accumulation in sequential pipelines or lack physical consistency by not explicitly using 3D geometry. GeoRelight addresses these limitations through two main technical innovations: isotropic NDC-Orthographic Depth (iNOD), a distortion-free 3D representation compatible with latent diffusion models, and a strategic mixed-data training approach that integrates both synthetic and auto-labeled real data. This joint optimization of geometry and relighting enables GeoRelight to outperform prior sequential models and systems that did not leverage 3D geometry.

Key takeaway

For research scientists developing computer vision models for human relighting or 3D reconstruction, GeoRelight demonstrates that a unified, joint approach significantly improves performance and physical consistency. You should consider integrating 3D geometry explicitly into your relighting pipelines and explore distortion-free 3D representations like iNOD to enhance latent diffusion model compatibility. This method offers a path to overcome limitations of sequential processing and geometry-agnostic systems.

Key insights

Jointly solving 3D geometry and relighting from a single image improves physical consistency and performance.

Principles

• 3D geometry and relighting are mutually beneficial tasks.
• Distortion-free 3D representations enhance latent diffusion models.

Method

GeoRelight uses a Multi-Modal Diffusion Transformer with isotropic NDC-Orthographic Depth (iNOD) and a mixed-data training strategy combining synthetic and auto-labeled real data to jointly reconstruct geometry and relight.

In practice

• Use iNOD for 3D representation in latent diffusion.
• Combine synthetic and real data for robust training.

Topics

• Geometrical Relighting
• 3D Reconstruction
• Multi-Modal Diffusion Transformers
• Isotropic NDC-Orthographic Depth
• Latent Diffusion Models

Best for: Research Scientist, AI Scientist, Computer Vision Engineer, Machine Learning Engineer

Related on AIssential

• Pixel Cube: Diffusion-based Portrait Video Relighting Through Realistic Lighting Reproduction — Diffusion models, trained on hybrid real/synthetic data from a specialized LED stage, enable photorealistic, temporally consistent portrait video relighting.
• GeoSAM-3D: Geodesic Prompt Propagation for Open-Vocabulary 3D Scene Segmentation from Monocular Video — GeoSAM-3D propagates 3D segmentation prompts using geodesic heat-kernel distance on Gaussian splatting scenes from monocular video, improving mask continuity and reducing leakage.
• IVGT: Implicit Visual Geometry Transformer for Neural Scene Representation — IVGT implicitly models continuous 3D geometry from unposed multi-view images using a neural scene representation.
• Modality Forcing for Scalable Spatial Generation — Modality Forcing enables scalable, joint image-depth generation using sparse data and T2I models, outperforming prior methods.
• PanoPlane: Plane-Aware Panoramic Completion for Sparse-View Indoor 3D Gaussian Splatting — Panoramic scene completion with layout-anchored attention steering significantly improves sparse-view 3D reconstruction.
• Latent Spatial Memory for Video World Models — Latent spatial memory in diffusion models significantly boosts video generation speed and reduces memory by avoiding pixel-space reconstruction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.