FADRA: Frequency-Aware Diffusion with Residual Adaptation for Video Face Restoration
Summary
FADRA, a novel frequency-aware diffusion framework, addresses the challenge of balancing spatial fidelity and temporal coherence in video face restoration (VFR) from severely degraded sequences. It adapts a frozen text-to-video diffusion model, Wan2.1-T2V-1.3B, using lightweight LoRA adapters and a Low-Quality Pixel-Alignment Feature Fusion module. A key innovation is the Repeated Residual Adaptation Head (RRAH), which performs step-wise residual refinement guided by low-quality cues and current velocity predictions. Additionally, a Frequency-Aware Loss (FAL) applies explicit supervision across spectral bands, emphasizing perceptually important facial details using a human visual system-inspired luminance prior. Evaluated on VFHQ and CelebV-HQ datasets, FADRA achieves superior quantitative metrics, including 29.95 PSNR and 38.97 FVD on VFHQ, and 28.70 PSNR and 48.32 FVD on CelebV-HQ, outperforming state-of-the-art methods. It also demonstrates higher inference efficiency at 0.866 FPS on an NVIDIA A100 and robustness to pose variations.
Key takeaway
For machine learning engineers developing video face restoration solutions, FADRA presents a compelling architecture to overcome the fidelity-coherence trade-off. You should investigate its Repeated Residual Adaptation Head and Frequency-Aware Loss components, as they demonstrably improve detail recovery and temporal stability. Implementing similar iterative, LQ-guided refinement and HVS-inspired frequency supervision can significantly enhance your models' perceptual quality and robustness, even under severe real-world degradations, while maintaining competitive inference speeds.
Key insights
FADRA enhances video face restoration by adapting a text-to-video diffusion model with iterative residual refinement and frequency-aware loss.
Principles
- Lightweight adaptation modules preserve strong generative priors for specialized tasks.
- Iterative, LQ-guided residual refinement improves detail recovery in generative processes.
- HVS-inspired frequency-domain loss enhances perceptual quality and structural integrity.
Method
FADRA adapts a frozen text-to-video diffusion model using LoRA and LQ feature fusion, then refines velocity predictions with a Repeated Residual Adaptation Head (RRAH) and applies a Frequency-Aware Loss (FAL) in the latent frequency domain.
In practice
- Employ LoRA for efficient fine-tuning of large generative models.
- Integrate low-quality input cues iteratively in diffusion model refinement.
- Consider frequency-aware losses with HVS priors for image/video quality tasks.
Topics
- Video Face Restoration
- Diffusion Models
- LoRA
- Frequency-Aware Loss
- Temporal Consistency
- Generative AI
Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer
Related on AIssential
Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.