BodyReLux: Temporally Consistent Full-Body Video Relighting

2026-01-04 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Gaming & Interactive Media · Depth: Expert, extended

Summary

BodyReLux is a novel, subject-specific video diffusion-based framework designed for temporally consistent full-body human performance relighting. It leverages a hybrid dataset, combining traditional static One-Light-at-a-Time (OLAT) captures with a new dynamic bi-packed performance capture method that rapidly interleaves two smoothly varying lighting sequences above the human flicker-fusion threshold, ensuring subject comfort. The model, finetuned from a pretrained text-to-video diffusion model (WAN2.2 5B), introduces OLAToken for accurate lighting control, representing each light source as a token, and uses masked attention for dynamic lighting. This approach achieves photorealistic, robust, and temporally consistent relighting for various framings, resolutions, and frame lengths, including casually captured videos, and generalizes well to in-the-wild scenarios. Training involved 8 NVIDIA A100 GPUs for 100K iterations over approximately 3 days.

Key takeaway

For creative technologists and post-production supervisors aiming to enhance human performance videos, BodyReLux offers a powerful solution to decouple lighting from capture constraints. You can achieve photorealistic, temporally consistent relighting of full-body performances, even with dynamic lighting changes, significantly reducing on-set lighting complexities. Consider integrating this diffusion-based approach to iterate on lighting designs in post-production, transforming artistic storytelling possibilities and potentially saving substantial production time and resources.

Key insights

BodyReLux enables photorealistic, temporally consistent full-body video relighting using a hybrid data capture and diffusion model with tokenized lighting control.

Principles

• Hybrid data (static OLAT + dynamic bi-pack) is crucial for diverse lighting and poses.
• Representing light sources as permutation-invariant tokens improves lighting control accuracy.
• Pretrained video diffusion models provide strong generative priors for photorealistic results.

Method

The method involves capturing static OLAT and dynamic bi-packed video data in an LED Sphere, preprocessing to create pixel-aligned video pairs, and finetuning a WAN2.2 5B video diffusion model with OLAToken and dynamic lighting attention for video-to-video relighting.

In practice

• Use bi-packed lighting sequences for comfortable, flicker-free dynamic video capture.
• Implement OLAToken for granular, compositional control over individual light sources.
• Apply block diagonal attention masks for precise frame-wise dynamic lighting control.

Topics

• Video Relighting
• Diffusion Models
• Full-Body Performance
• Lighting Conditioning
• Data Capture
• Post-production

Code references

• modelscope/DiffSynth-Studio
• google-research/frame-interpolation

Best for: Research Scientist, AI Scientist, Computer Vision Engineer, Creative Technologist

Related on AIssential

• Pixel Cube: Diffusion-based Portrait Video Relighting Through Realistic Lighting Reproduction — Diffusion models, trained on hybrid real/synthetic data from a specialized LED stage, enable photorealistic, temporally consistent portrait video relighting.
• DuET: Dual Expert Trajectories for Diffusion Image Editing — DuET enhances diffusion image editing by temporarily relaxing source-image conditioning through a text-to-image phase, improving edit fidelity without added cost.
• GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers — Jointly solving 3D geometry and relighting from a single image improves physical consistency and performance.
• Real2SAM2Real: Generative 3D Caches as Complementary Context for Video Diffusion — Real2SAM2Real uses editable 3D caches from lifting models to provide robust geometric scaffolds for Video Diffusion Models, improving control and consistency.
• Understanding Identity Continuity in Thermal Video through Scene-Level Consistency — High-precision trajectory relinking using scene-level consistency effectively recovers identity in thermal video MOT.
• MotionEnhancer: Leveraging Video Diffusion for Motion-Enhanced Vision-Language Models — Video Diffusion Model attention provides motion-calibrated priors to enhance Vision-Language Model motion understanding via attention alignment.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.