DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoders

2026-05-21 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

DecQ is a novel framework designed to enhance Representation Autoencoders (RAEs) by resolving the inherent trade-off between reconstruction quality and generative fidelity. RAEs typically leverage frozen vision foundation models (VFMs) as tokenizer encoders, which aids fast convergence and high-quality generation in latent diffusion models but constrains fine-grained spatial reconstruction. Conversely, fine-tuning VFMs for reconstruction degrades their pretrained semantic space and generative performance. DecQ introduces lightweight detail-condensing queries that extract fine-grained information from intermediate VFM features using condenser modules. These queries are incorporated into the decoder for improved reconstruction and are jointly generated with patch tokens during generative modeling. This approach, aggregating information from both shallow and deep layers, significantly improves performance. Experiments show DecQ, with only 8 additional queries and 3.9% extra computation, boosts PSNR from 19.13 dB to 22.76 dB over a frozen DINOv2-based RAE. It also achieves 3.3x faster convergence for generative modeling, reaching an FID of 1.41 without guidance and 1.05 with guidance.

Key takeaway

For Machine Learning Engineers developing latent diffusion models or representation autoencoders, DecQ offers a compelling solution to the reconstruction-generation trade-off. You should consider integrating detail-condensing queries to significantly boost reconstruction PSNR from 19.13 dB to 22.76 dB and achieve 3.3x faster generative model convergence. This approach allows you to enhance fine-grained image editing and generation without compromising semantic fidelity, using minimal additional computation.

Key insights

DecQ uses detail-condensing queries to balance reconstruction and generation in RAEs, improving both with minimal overhead.

Principles

• Freezing VFMs in RAEs limits fine-grained reconstruction.
• Fine-tuning VFMs for reconstruction degrades generative fidelity.
• Aggregating shallow and deep VFM features enhances performance.

Method

DecQ introduces lightweight detail-condensing queries to extract fine-grained information from intermediate VFM features via condenser modules. These queries are integrated into the decoder for reconstruction and generated with patch tokens.

In practice

• Improve RAE reconstruction PSNR from 19.13 dB to 22.76 dB.
• Achieve 3.3x faster generative model convergence.
• Reduce FID to 1.05 with guidance in latent diffusion.

Topics

• Representation Autoencoders
• Latent Diffusion Models
• Vision Foundation Models
• Image Reconstruction
• Generative Modeling
• DINOv2

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Building Generative Image & Video models at Scale - Sander Dieleman (Veo and Nano Banana) — Diffusion models generate high-quality audiovisual data through iterative denoising of learned compressed latent representations.
• Distilling Drifting Transformers with Representation Autoencoders — Drift-RAE enables stable, high-performance distillation of flow models in semantically rich Representation Autoencoder latent spaces.
• Dual-Latent Collaborative Decoding for Fidelity-Perception Balanced Image Compression — Decomposing image compression into fidelity and perception experts via dual-latent decoding balances conflicting reconstruction goals.
• Generative Modeling of Complex-Valued Brain MRI Data — Jointly modeling MRI magnitude and phase data improves synthetic data generation and diagnostic classification.
• Pixel-Level Residual Diffusion Transformer: Scalable 3D CT Volume Generation — PRDiT uses a two-stage, coarse-to-fine diffusion transformer to generate high-resolution 3D medical volumes directly at voxel-level.
• Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster — Learning the forward noising process in discrete diffusion enables few-step, high-quality generation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.