Expert-aware quantisation: near-Q4 quality at near-Q2 size?

2026-06-22 · Source: Martin Alderson · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

Expert-aware quantization for Mixture-of-Experts (MoE) models proposes a method to achieve near-Q4 quality at near-Q2 model sizes by selectively quantizing experts based on task-specific usage. The research profiled Qwen3.6 35B-A3B on C++ programming tasks, observing that during code generation, the top 32 of 256 experts handled approximately 50% of the routing, indicating concentrated expert activity. By applying higher precision (e.g., Q4) to these "hot" experts and aggressive 2-bit quantization to "cold" experts, a Q4-hot / Q2-cold tiered model achieved perplexity scores of 1.635 (reading C++) and 1.477 (writing C++). This 14GB model, only 1GB larger than a uniform Q2 model (13GB, PPL 2.103), recovered about 90% of the quality gap between uniform Q2 and uniform Q4 (20GB, PPL 1.582), demonstrating significant quality improvement for smaller model sizes.

Key takeaway

For AI Engineers optimizing Mixture-of-Experts models for local or edge deployment, this research demonstrates a viable strategy to achieve near-Q4 quality at near-Q2 model sizes. By profiling expert usage for specific tasks and applying tiered quantization (e.g., Q4 for hot experts, Q2 for cold), you can significantly reduce model memory footprint without a proportional loss in performance. Explore integrating domain-specific expert profiling into your quantization workflows to maximize efficiency for targeted applications.

Key insights

Selectively quantizing MoE experts based on task-specific usage can achieve near-Q4 quality at near-Q2 model sizes.

Principles

• MoE expert usage is task-dependent.
• Prioritizing precision for "hot" experts improves quality.
• Profiling identifies critical experts for specific domains.

Method

Profile MoE models on specific tasks to identify "hot" and "cold" experts. Apply higher precision quantization (e.g., Q4) to hot experts and lower precision (e.g., Q2) to cold experts.

In practice

• Quantize MoE models for domain-specific deployment.
• Develop tools for expert-level profiling.
• Explore sub-2-bit quantization for cold experts.

Topics

• Mixture-of-Experts
• Model Quantization
• Expert Profiling
• Qwen3.6 35B-A3B
• llama.cpp
• Perplexity Evaluation

Code references

• ggml-org/llama.cpp

Best for: NLP Engineer, Research Scientist, AI Engineer, Machine Learning Engineer, AI Scientist

Related on AIssential

• Mixture of Experts (MoE) - More Parameters, Same Compute — Mixture of Experts scales model capacity by having many specialized networks, but keeps per-token compute low by activating only a few.
• Value-and-Structure Alignment for Routing-Consistent Quantization of Mixture-of-Experts Models — VSRAQ quantizes MoE models by aligning routing-relevant values and expert structure to prevent expert-selection instability.
• Autoresearching Apple's "LLM in a Flash" to run Qwen 397B locally — Efficient LLM inference on limited memory is possible by streaming expert weights from flash storage.
• Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models — Preserving cross-corpus expert coverage with generic data effectively prunes Sparse Mixture-of-Experts language models.
• How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle — A unified formulation provides a principle for selecting optimal MoE expert pruning criteria based on task requirements.
• Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts — Expert upcycling efficiently scales MoE models by duplicating experts mid-training, preserving inference cost and leveraging warm initialization.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Martin Alderson.