LightCrafter: PBR-Conditioned Video Diffusion Refinement for Controllable and Consistent Relighting

2026-07-09 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Graphics · Depth: Expert, quick

Summary

LightCrafter is a novel hybrid pipeline designed for controllable and consistent video relighting, addressing challenges in temporal consistency and physically grounded light transport. It reformulates video relighting as a video translation task, where a Physically-Based Rendering (PBR) of the input video under target illumination serves as a proxy. This approach bakes illumination targets into the PBR proxy, simplifying the diffusion model's learning of illumination concepts like environment maps, and inherently provides long-form temporal consistency. LightCrafter leverages photometric priors by post-training CogVideoX on synthetic video pairs and real-world unpaired videos to capture complex effects such as global illumination. The system outperforms prior state-of-the-art on existing real-world relighting benchmarks and introduces a new synthetic benchmark. Its dataset, metrics, and code will be released.

Key takeaway

For Computer Vision Engineers and Graphics Developers needing high-quality, controllable video relighting, LightCrafter offers a robust solution. You should consider its PBR-conditioned diffusion refinement approach to overcome limitations of traditional inverse rendering or direct generative methods, especially when global illumination effects are critical. This method simplifies illumination concept learning for diffusion models, streamlining development for complex visual effects and ensuring superior temporal consistency in your projects.

Key insights

LightCrafter conditions video diffusion models with PBR-rendered proxy videos for consistent, controllable relighting, simplifying illumination learning.

Principles

• Combine inverse rendering with generative video translation.
• PBR proxies embed illumination, guiding diffusion models.
• Post-training refines complex global illumination effects.

Method

Reformulates video relighting by translating a PBR rendering of the input under target illumination to the final target, then refines results using a post-trained CogVideoX model.

In practice

• Achieve intricate and temporally consistent video relighting.
• Enhance diffusion model control over lighting conditions.
• Utilize PBR outputs as effective conditioning for generative models.

Topics

• Video Relighting
• Diffusion Models
• Physically-Based Rendering
• Computer Vision
• Graphics
• Temporal Consistency
• Global Illumination

Best for: Research Scientist, AI Scientist, Computer Vision Engineer, Machine Learning Engineer

Related on AIssential

• Pixel Cube: Diffusion-based Portrait Video Relighting Through Realistic Lighting Reproduction — Diffusion models, trained on hybrid real/synthetic data from a specialized LED stage, enable photorealistic, temporally consistent portrait video relighting.
• PIXLRelight: Controllable Relighting via Intrinsic Conditioning — PIXLRelight enables physically accurate, controllable single-image relighting by bridging PBR and learned synthesis via intrinsic conditioning.
• Novel View Synthesis as Video Completion — Video diffusion models can be adapted for sparse novel view synthesis by treating it as low frame-rate video completion.
• FreeAnimate: Training-Free Human Image Animation with Preview-Guided Denoising — FreeAnimate enables high-quality, training-free human image animation by employing preview-guided denoising and attention mechanisms in diffusion models.
• PIXLRelight: Controllable Relighting via Intrinsic Conditioning — PIXLRelight enables fast, high-quality single-image relighting by integrating PBR with learned image synthesis via intrinsic conditioning.
• DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models — DiffusionOPD unifies multi-task diffusion model training via online policy distillation, improving efficiency and performance.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.