VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizers

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

VQ-Transplant is a novel framework designed to efficiently integrate new Vector Quantization (VQ) modules into frozen, pre-trained visual tokenizers, such as VAR, without requiring costly end-to-end retraining. This plug-and-play approach preserves existing encoder-decoder parameters, significantly reducing computational overhead. It addresses decoder-quantization mismatch through a lightweight decoder adaptation strategy, trained for only 5 epochs on ImageNet-1k. The framework, along with the proposed MMD VQ method, achieves near state-of-the-art reconstruction fidelity for industry-level models like VAR, while cutting training costs by 95%. This democratizes VQ research by enabling resource-efficient exploration of novel quantization techniques.

Key takeaway

For AI Scientists and Machine Learning Engineers developing novel VQ algorithms, VQ-Transplant offers a critical pathway to accelerate research and deployment. You can integrate new VQ modules into pre-trained visual tokenizers like VAR, achieving near state-of-the-art reconstruction fidelity with a 95% reduction in training cost. This enables rapid iteration and experimentation, making advanced quantization techniques accessible even under resource constraints. Consider adopting this two-stage framework to significantly cut development cycles.

Key insights

VQ-Transplant decouples VQ module development from costly full tokenizer retraining, enabling efficient innovation.

Principles

• Preserve frozen encoder-decoder parameters to save computational cost.
• Align feature and codebook distributions for optimal VQ performance.
• Lightweight decoder adaptation mitigates quantization space mismatch.

Method

VQ-Transplant involves two stages: first, substituting the native VQ module with a new one while freezing encoder-decoder, then performing lightweight decoder adaptation for 5 epochs to align feature priors.

In practice

• Integrate MMD VQ for robust distribution-aligned quantization.
• Apply 5-epoch decoder adaptation on ImageNet-1k to restore fidelity.
• Utilize pre-trained VAR tokenizers as a base for VQ module experimentation.

Topics

• Vector Quantization
• Visual Tokenization
• Model Adaptation
• Computational Efficiency
• MMD VQ
• Image Reconstruction
• Pre-trained Models

Code references

• VQ-Research/VQ-Transplant

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizers — VQ-Transplant enables efficient, plug-and-play integration of new VQ modules into pre-trained visual tokenizers, drastically cutting retraining costs.
• Moving Beyond Diversity: Visual Token Pruning as Subspace Reconstruction for Efficient VLMs — SPARE prunes VLM visual tokens by minimizing reconstruction error and leveraging anti-relevance for context preservation.
• Modular Monolingual Adaptation using Pretrained Language Models — Freezing embeddings during monolingual adaptation of PLMs for low-resource languages prevents overfitting and improves performance.
• Variable-Length Tokenization via Learnable Global Merging for Diffusion Transformers — Merging similar tokens enables variable-length tokenization while preserving cross-length representation alignment in Diffusion Transformers.
• The Hidden Evolution of Disguised Visual Context inside the VLM — Visual token integration paradigms in VLMs dictate internal representation evolution, affecting feature utilization and task performance.
• Beyond Attention Scores: SVD-Based Vision Token Pruning for Efficient Vision-Language Models — SVD-Prune uses global variance and leverage scores for training-free VLM token pruning, outperforming attention-based methods.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.