Bringing Nunchaku 4-bit Diffusion Inference to Diffusers

2026-07-08 · Source: Hugging Face - Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, long

Summary

Hugging Face Diffusers now natively supports Nunchaku 4-bit diffusion inference, a significant advancement published on July 23, 2026, for running large diffusion transformers on consumer GPUs. Traditional BF16 precision models often demand 20-30 GB of VRAM, but Nunchaku's SVDQuant method employs 4-bit weights and activations (W4A4) to reduce memory usage and accelerate the denoising process. The new Nunchaku Lite integration allows users to load pre-quantized checkpoints directly via `from_pretrained()` without local CUDA compilation, leveraging the `kernels` package. This integration delivers approximately a 30% speedup and up to 50% VRAM reduction, exemplified by an ERNIE-Image-Turbo model generating a 1024x1024 image in 1.7 seconds with 12 GB peak memory on an RTX 5090. Further optimizations like `torch.compile` can boost speedup to 1.8x, while the `diffuse-compressor` toolkit enables users to quantize new models.

Key takeaway

For AI Engineers deploying large diffusion models on consumer GPUs, Nunchaku Lite integration into Diffusers fundamentally changes hardware accessibility. You can now achieve up to 1.8x faster inference and 50% VRAM reduction. This 4-bit W4A4 quantization makes models like ERNIE-Image-Turbo viable on systems with 12 GB VRAM. Consider adopting pre-quantized checkpoints or using `diffuse-compressor` to optimize your custom architectures for broader deployment.

Key insights

Nunchaku Lite integrates 4-bit W4A4 quantization into Diffusers, enabling faster, lower-VRAM diffusion model inference on consumer GPUs.

Principles

• SVDQuant handles activation outliers by moving them to weights.
• Fused kernels eliminate memory access overhead for 4-bit paths.
• Quantizing both weights and activations improves speed and memory.

Method

Nunchaku Lite patches `nn.Linear` modules with SVDQ/AWQ layers, loading CUDA kernels from the `kernels` package for 4-bit W4A4 and W4A16 quantization.

In practice

• Load pre-quantized models using `pipe.from_pretrained()`.
• Combine with `torch.compile` for 1.8x speedup.
• Use `diffuse-compressor` to quantize new architectures.

Topics

• Diffusion Models
• Model Quantization
• Nunchaku Lite
• Hugging Face Diffusers
• SVDQuant
• GPU Memory Optimization

Code references

• huggingface/blog
• nunchaku-tech/nunchaku
• huggingface/kernels
• rootonchair/diffuse-compressor
• nunchaku-ai/nunchaku

Best for: Machine Learning Engineer, AI Engineer, MLOps Engineer

Related on AIssential

• FourTune: Towards Fully 4-Bit Efficient Post-Training for Diffusion Models — FourTune enables stable, efficient 4-bit post-training for large diffusion models by isolating outliers and optimizing low-bit computation.
• Accelerating Diffusers and xDiT Image Generation with MXFP4 using AMD Quark on AMD Instinct™ MI350 GPUs — AMD Quark's MXFP4 quantization significantly boosts diffusion model inference speed on MI350 GPUs with minimal quality loss.
• Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncation — DMLLMs reveal output boundaries early via MLP sparsity, enabling efficient, training-free truncation.
• Open-Source AI Tool Upgrades Speed Up LLM and Diffusion Models on NVIDIA RTX PCs — NVIDIA's CES 2026 announcements significantly boost AI PC developer capabilities through framework optimizations and new model releases.
• TileFuse: A Fused Mixed-Precision Kernel Library for Efficient Quantized LLM Inference on AMD NPUs — TileFuse enables efficient AWQ-style quantized LLM inference on AMD XDNA2 NPUs through fused mixed-precision kernels.
• KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformers — KroQuant uses Kronecker-structured block transforms and LoRaQ calibration to enable efficient W4A4 post-training quantization for Diffusion Transformers.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Hugging Face - Blog.