Rethinking Token Reduction for Diffusion Models via Output-Similarity-Awareness

2026-05-22 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, extended

Summary

DiTo, a novel Diffusion-specific Token Reduction paradigm, addresses the quadratic computational complexity of Diffusion Transformers (DiTs) by shifting to output-centric token reduction. Existing methods, which rely on input token similarity, often lead to significant recovery errors and quality degradation in generative tasks. DiTo minimizes these errors by leveraging the temporal consistency of Diffusion Models, using prior-step output similarities as a proxy to establish token correspondences during a "Matching step." These correspondences are then efficiently reused across multiple "Reduction steps." The method incorporates PMR-guided Interval Scheduling to optimize matching frequency and Frequency-aware Token Matching with a selection-frequency penalty to mitigate localized approximation errors and blocking artifacts from repeated reuse. Experiments on FLUX.1-dev (35 steps) and SD3 (50 steps) at 1024x1024 resolution show DiTo consistently outperforms prior methods, achieving 1.6–3.9 dB higher PSNR and up to an 18.6% latency reduction (e.g., 21.69s on an RTX 6000 Ada GPU), demonstrating a superior quality-efficiency trade-off.

Key takeaway

For Machine Learning Engineers optimizing Diffusion Transformer inference, your current token reduction strategy likely sacrifices image quality. You should shift to output-centric token reduction, leveraging prior-step output similarities to minimize recovery error. Implement dynamic interval scheduling and frequency-aware token matching to achieve superior visual fidelity and structural consistency, as DiTo demonstrates 1.6–3.9 dB PSNR gains with comparable speedups. This approach significantly improves the quality-efficiency trade-off for high-resolution generative tasks.

Key insights

Output-centric token reduction in Diffusion Models, guided by prior-step output similarity, minimizes recovery error and boosts generation quality.

Principles

• Generative token reduction must reflect output similarity.
• DM token similarity is consistent across adjacent timesteps.
• Reuse token correspondences for efficiency.

Method

DiTo decouples token matching (using prior output) from reduction/recovery, employing PMR-guided interval scheduling and frequency-aware token matching to optimize reuse and mitigate artifacts.

In practice

• Prioritize output similarity for generative model token reduction.
• Leverage temporal consistency to reuse matching results.
• Apply selection frequency penalties to prevent artifacts.

Topics

• Diffusion Transformers
• Token Reduction
• Image Synthesis
• Computational Efficiency
• Output Similarity
• PMR Scheduling

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Rethinking Token Reduction for Diffusion Models via Output-Similarity-Awareness — DiTo improves Diffusion Transformer efficiency by focusing token reduction on output similarity, not just input similarity.
• Variable-Length Tokenization via Learnable Global Merging for Diffusion Transformers — Merging similar tokens enables variable-length tokenization while preserving cross-length representation alignment in Diffusion Transformers.
• Building Generative Image & Video models at Scale - Sander Dieleman (Veo and Nano Banana) — Diffusion models generate high-quality audiovisual data through iterative denoising of learned compressed latent representations.
• PTL-Diffusion: Manifold-Aware Diffusion with Periodic Terminal Laws — PTL-Diffusion improves manifold data generation by using periodic, structured terminal laws in its forward noising process.
• Continuous Language Diffusion as a Decoder-Interface Problem — Continuous language diffusion models succeed by guiding latent states into "decoder-basin" regions where decoders can reliably interpret tokens.
• InsightTok: Improving Text and Face Fidelity in Discrete Tokenization for Autoregressive Image Generation — Localized, content-aware perceptual losses significantly enhance text and face fidelity in discrete image tokenization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.