Keep The Essentials: Efficient Reference Conditioned Generation via Token Dropping

2026-06-22 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, medium

Summary

Reference-based diffusion models, which guide prompt-driven image synthesis using elements from input images, face significant computational expense, especially with multiple references. While efficiency in prompt-driven generation is explored, reference-based models present unique challenges due to wasteful dense token grid representations. Researchers introduce Sparse Context, a method to create sparse reference representations by retaining a reduced subset of tokens. They found that dropping a significant portion of tokens at inference time largely preserves generation capabilities, even without model modification. Fine-tuning with random token dropping at varying ratios enhances robustness to partial representations, decoupling the model from specific selection rules. At inference, task-aware token selection prioritizes informative regions, adapting the token budget. This approach achieves a 4x increase in inference speed for multi-reference generation and 2x for single-reference generation, without compromising visual quality in spatially-aligned editing or subject-driven generation.

Key takeaway

For Machine Learning Engineers optimizing reference-based diffusion models, adopting Sparse Context offers a direct path to significant inference speedups. You can achieve a 4x speed increase for multi-reference and 2x for single-reference generation without compromising visual quality. Consider integrating this token dropping and fine-tuning strategy to enhance your model's efficiency and deployment, especially when working with computationally intensive reference-based tasks.

Key insights

Sparse Context significantly boosts reference-based diffusion model inference speed by intelligently dropping non-essential reference tokens.

Principles

• Reference token sparsity can maintain generation quality.
• Random token dropping during training enhances model robustness.

Method

Sparse Context constructs sparse reference representations by retaining a reduced subset of tokens, fine-tuning with random dropping, and applying task-aware selection at inference time.

In practice

• Achieve 4x inference speed for multi-reference generation.
• Gain 2x speed for single-reference generation without quality loss.

Topics

• Reference-based Diffusion Models
• Token Dropping
• Inference Optimization
• Sparse Context
• Image Generation
• Computational Efficiency

Code references

• FloatAI/TKEval
• HaozheZhao/MENTOR

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Semantic-Aware Prefix Learning for Token-Efficient Image Generation — SMAP improves image generation by making semantic conditions functionally necessary for latent representation learning.
• Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation — Sparse Forcing improves long-horizon video generation quality and efficiency by leveraging trainable sparse attention and persistent memory.
• Mean-Field Parallel Decoding for Discrete Diffusion Language Models — A training-free framework coordinates parallel token updates in discrete diffusion models, improving generation quality and latency.
• How Much Dense Attention is Necessary? Oracle-Guided Sparse Prefill for Full/GQA Layers in Hybrid Long-Context Models — The study quantifies dense attention needs for long-context models, achieving significant speedups with minimal quality loss via sparse prefill.
• TAPS: Target-Aware Prefix Tree Selection for Diffusion-Drafted Speculative Decoding — TAPS optimizes speculative decoding by using target-aware prefix selection to improve acceptance-cost tradeoffs in diffusion-drafted trees.
• Remask, Don't Replace: Token-to-Mask Refinement in Masked Diffusion Language Models — Remasking erroneous tokens to [M] improves masked diffusion language model accuracy by providing a neutral context for re-prediction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.