Diet-KIT: Post-Training Quantization for Speech LLMs

2026-06-29 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

Diet-KIT is a system designed for the IWSLT speech translation compression task, aiming to reduce the 16 GB Qwen2-Audio-7B base model to under a strict 4 GB on-disk storage constraint. It employs a sequential pipeline based on Half-Quadratic Quantization (HQQ). Systematic ablations revealed that 4-bit quantization largely preserves translation quality, while 3-bit quantization causes a significant performance drop, preventing its uniform application. The system found that the embedding table tolerates 2-bit quantization with minimal loss, but the LM head requires higher precision. To meet the storage target, Diet-KIT uses a sensitivity-guided layer selection method, identifying MLP sublayers suitable for 3-bit compression through per-layer sensitivity analysis, which outperformed manual and random selection. AWQ calibration refines the process. The final system achieves 3.98 GB on disk, with COMET scores of 74.4 on en→de and 77.1 on en→zh, compared to 75.6 and 79.5 for the uncompressed model.

Key takeaway

For Machine Learning Engineers optimizing speech LLMs for strict deployment constraints, Diet-KIT demonstrates a viable path to significant model compression. You should consider a mixed-precision quantization strategy, starting with 4-bit for general layers and 2-bit for embedding tables. Apply sensitivity-guided analysis to identify specific MLP sublayers that tolerate 3-bit compression. This avoids severe performance cliffs, enabling models like Qwen2-Audio-7B to fit within a 4 GB footprint.

Key insights

Diet-KIT achieves significant speech LLM compression by selectively applying quantization based on component sensitivity.

Principles

• 4-bit quantization maintains quality; 3-bit often causes sharp performance drops.
• Embedding tables tolerate aggressive 2-bit quantization.
• LM heads require higher precision than other components.

Method

A sequential pipeline uses HQQ, followed by sensitivity-guided layer selection for 3-bit MLP compression, and finally AWQ calibration for refinement.

In practice

• Quantize embedding tables to 2-bit for maximum space savings.
• Prioritize 4-bit quantization for most layers, avoiding aggressive 3-bit globally.
• Use sensitivity analysis to identify specific MLP sublayers for 3-bit compression.

Topics

• Post-Training Quantization
• Speech LLMs
• Model Compression
• Half-Quadratic Quantization
• Sensitivity Analysis
• Qwen2-Audio-7B
• IWSLT

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Making Large Language Models Smaller, Faster, and Practical: A Friendly Guide to Quantization — Model quantization compresses LLMs by converting high-precision weights to lower-precision formats, enhancing efficiency.
• Leveraging Audio-LLMs to Filter Speech-to-Speech Training Data — An audio-LLM can effectively filter noisy speech-to-speech translation data using a Rank-to-Distill strategy, improving S2ST performance.
• ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization — ScaleSweep optimizes NVFP4 quantization by sweeping block scale candidates within derived theoretical bounds to minimize reconstruction error.
• Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clustering — Parameter clustering offers data-free, training-free, and hardware-friendly compression for speech foundation models, outperforming magnitude-based pruning.
• Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression — Jointly optimizing structural pruning and mixed-precision quantization globally improves LLM compression and performance at ultra-low bitrates.
• TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization — TWLA enables 1.58-bit weights and 4-bit activations for LLMs, overcoming heavy-tailed activation challenges for end-to-end acceleration.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.