PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

2026-07-16 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, AI Hardware Optimization · Depth: Expert, quick

Summary

PolyQ is a novel CPU-oriented compiler/quantization co-design framework designed for scalable edge CPU LLM inference. It addresses the limitations of existing low-bit quantization methods on CPUs by enabling activation-aware, channel-wise bit allocation under a user-specified average-bit budget, using bit-widths from {2,3,4,8,16}. PolyQ's compile-time model compiler permutes and clusters channels into bit-homogeneous blocks, generates SIMD- and LUT-compatible kernels, and merges compatible permutations to optimize runtime efficiency. This approach facilitates practical fractional-bit deployment for CPU-only inference. Benchmarking on Falcon-H1-3B, Llama2-13B, and Qwen3-32B on WikiText-2 shows PolyQ improves perplexity by 2.4-32.1% over prior methods at a 3-bit target. End-to-end measurements on workstation, laptop, and mobile CPUs demonstrate up to 70.8% reduction in activation reorder traffic, proportional scaling of prefill latency and decode throughput, and energy/token overhead below 2%.

Key takeaway

For Machine Learning Engineers optimizing LLM deployment on edge CPUs, PolyQ offers a practical path to fractional-bit quantization. You should consider this co-design approach to achieve significant perplexity improvements (2.4-32.1% at 3-bit) and reduce activation reorder traffic by up to 70.8%. This enables more efficient and predictable LLM inference on diverse CPU targets, including mobile devices, without substantial energy overhead.

Key insights

PolyQ enables practical, efficient fractional-bit LLM inference on diverse edge CPUs through compiler/quantization co-design.

Principles

• Co-designing compiler and quantization optimizes CPU efficiency.
• Channel-wise bit allocation improves low-bit LLM quality.
• Layout regularization reduces activation reorder traffic.

Method

PolyQ assigns per-channel bit-widths from {2,3,4,8,16} then uses a compile-time model compiler to permute, cluster channels, and generate SIMD/LUT kernels, merging permutations for efficient CPU execution.

In practice

• Deploy fractional-bit LLMs on edge CPUs.
• Achieve 2.4-32.1% perplexity gains at 3-bit.
• Reduce activation reorder traffic by up to 70.8%.

Topics

• LLM Inference
• Edge Computing
• CPU Optimization
• Quantization
• Compiler Co-design
• Fractional-bit Quantization

Best for: NLP Engineer, AI Scientist, Research Scientist, AI Hardware Engineer, Machine Learning Engineer, AI Engineer

Related on AIssential

• Channel-Wise Mixed-Precision Quantization for Large Language Models — CMPQ adaptively quantizes LLMs channel-wise using mixed-precision and outlier protection for efficient edge deployment.
• ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM — ExaGEMM enables efficient low-bit GEMM on CPUs via in-register associative computing and workload-aware codesign.
• TileFuse: A Fused Mixed-Precision Kernel Library for Efficient Quantized LLM Inference on AMD NPUs — TileFuse enables efficient AWQ-style quantized LLM inference on AMD XDNA2 NPUs through fused mixed-precision kernels.
• AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization — Adaptive codebooks, learned via activation-weighted k-means, significantly improve 4-bit LLM quantization without high computational cost.
• KronQ: LLM Quantization via Kronecker-Factored Hessian — KronQ improves LLM post-training quantization by incorporating gradient covariance via Kronecker-factored Hessian approximation.
• How I Fit a Model That “Shouldn’t” Fit on a 6GB Laptop GPU — Quantization reduces LLM memory footprint by storing weights less precisely, enabling deployment on resource-constrained GPUs.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.