SGMD: Score Gradient Matching Distillation for Few-Step Video Diffusion Distillation

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

Score Gradient Matching Distillation (SGMD) is a novel method for accelerating few-step video diffusion models, addressing key limitations of existing Distribution Matching Distillation (DMD) approaches. DMD-style methods face challenges with high training costs due to continuously tracking an evolving generator and conservative reverse-KL matching that can suppress motion dynamics. SGMD tackles this by directly optimizing the fake score towards the teacher using a stable teacher stop-gradient Fisher objective. It employs dual potentials—negative-residual (NR) for generator updates and residual-contraction (RC) for fake-score updates—to decouple outer-loop correction from inner-loop tracking. Benchmarked against DMD2 using the Wan2.1-T2V-14B teacher model, SGMD demonstrated an approximate ~3x training speedup and substantially improved motion dynamics for 4-step distilled models, while maintaining temporal consistency. Human evaluations confirmed SGMD's superiority in motion quality and overall preference, with visual quality and text alignment remaining comparable.

Key takeaway

For Machine Learning Engineers optimizing video diffusion models for faster inference, SGMD offers a compelling alternative to DMD-style methods. You should consider implementing SGMD to achieve an approximate 3x training speedup and significantly enhance motion dynamics in 4-step distilled models. Experiment with a moderate λ value, such as 0.1, to balance distribution matching and tracking correction, ensuring stable training and superior perceptual quality.

Key insights

SGMD accelerates video diffusion distillation by directly optimizing fake scores with stable Fisher objectives and dual potentials.

Principles

• Teacher stop-gradient Fisher provides stable distribution matching.
• Decoupling outer-loop correction from inner-loop tracking improves stability.
• Reverse-KL objectives can suppress motion in few-step video generation.

Method

SGMD uses a two-step update: generator updates with Fisher objective + negative-residual (NR) potential, then fake score updates with residual-contraction (RC) potential.

In practice

• Reduce fake-score updates per iteration from 5 to 1 for ~3x speedup.
• Use a moderate λ (e.g., 0.1) to balance tracking and distribution matching.
• Apply SGMD for improved motion dynamics in 4-step video distillation.

Topics

• Video Diffusion Models
• Model Distillation
• Few-Step Inference
• Score Matching
• Training Acceleration
• Motion Dynamics

Code references

• ModelTC/LightX2V

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• SGMD: Score Gradient Matching Distillation for Few-Step Video Diffusion Distillation — SGMD directly optimizes fake scores with stable teacher stop-gradient Fisher for faster, better video distillation.
• Initialization-Aware Score-Based Diffusion Sampling — Optimizing the backward process initialization in SGMs significantly reduces computational cost while maintaining generative quality.
• stochastic gradient descent #maths #statistics #datascience #machinelearning #dataanlysis — Faster iteration with noisy gradients often outperforms slow, precise convergence in machine learning.
• Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation — On-Policy Distillation updates are sparse and geometrically distinct, preserving unique signatures despite dense teacher supervision.
• Optimal Self-Distillation for Rectified Flow via Linear Probing — Optimal self-distillation provably improves rectified flow models by deriving and applying an optimal mixing coefficient for teacher and true velocities.
• Drive-KD: Multi-Teacher Distillation for VLMs in Autonomous Driving — Small VLMs can achieve superior autonomous driving performance and efficiency through multi-teacher knowledge distillation and targeted attention signals.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.