Quant.npu: Enabling Efficient Mobile NPU Inference for on-device LLMs via Fully Static Quantization

2026-01-29 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Internet of Things (IoT) & Connected Devices · Depth: Expert, extended

Summary

Quant.npu is an integer-only fully static quantization framework designed for efficient on-device Large Language Model (LLM) inference on mobile Neural Processing Units (NPUs). It addresses the incompatibility of existing post-training quantization methods, which rely on dynamic activation quantization, with NPU hardware constraints. Quant.npu integrates learnable quantization parameters and rotation matrices, employing a rotation-and-bit-width-aware initialization, a two-stage distribution-aware selective optimization pipeline, and a sensitivity-guided adaptive mixed-precision scheme. Evaluated on Qualcomm SM8650 and SM8750 NPUs, Quant.npu achieves comparable accuracy to leading methods while reducing inference latency by up to 15.1% for models such as Llama-3.2-3B-Instruct and Qwen2.5-3B-Instruct, demonstrating broad applicability and strong generalization.

Key takeaway

For Machine Learning Engineers deploying LLMs on mobile NPUs, Quant.npu provides a critical framework to achieve efficient, low-latency inference without significant accuracy loss. You should consider its rotation-and-bit-width-aware initialization and two-stage selective optimization to overcome the challenges of fully static quantization. This approach enables substantial speedups, up to 15.1%, making it highly suitable for power-constrained edge deployments.

Key insights

Quant.npu bridges high-fidelity PTQ and NPU-constrained inference via integer-only fully static quantization with learnable parameters and rotations, achieving leading accuracy.

Principles

• Mobile NPUs demand fully static, integer-only quantization.
• Quantization parameter initialization critically impacts optimization stability.
• Orthogonal rotations smooth activation distributions.

Method

Quant.npu employs a two-stage pipeline: gradient-based optimization for rotated distributions and static calibration for unrotated tensors, complemented by adaptive mixed-precision for sensitive layers.

In practice

• Initialize rotated activations with Mean-based, unrotated with Max-Min.
• Decouple optimization for rotated and unrotated tensor distributions.
• Promote top 10% sensitive down_proj activations to 16-bit.

Topics

• Mobile NPU Inference
• Fully Static Quantization
• Large Language Models
• Rotation-based Quantization
• Mixed-Precision Quantization
• On-device AI

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs — CAT-Q enables accurate, cost-efficient ternary quantization for LLMs via post-training methods, significantly reducing training data and time.
• TileFuse: A Fused Mixed-Precision Kernel Library for Efficient Quantized LLM Inference on AMD NPUs — TileFuse enables efficient AWQ-style quantized LLM inference on AMD XDNA2 NPUs through fused mixed-precision kernels.
• Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats — HiFloat formats, particularly HiF4, offer robust low-bit quantization for LLM inference on Ascend NPUs, especially at 4-bit precision.
• ReQAT: Achieving Full-Precision Reasoning Accuracy with 4-bit Floating-Point Quantization-Aware Training — ReQAT enables 4-bit quantization for LRMs, recovering full-precision reasoning accuracy by targeting low-entropy token sensitivity.
• Gemma 4 QAT models: Optimizing model compression for mobile and laptop efficiency — Quantization-Aware Training (QAT) significantly reduces LLM memory footprint while preserving quality for on-device deployment.
• Cascaded Multi-Granularity Pruning for On-Device LLM Inference in Industrial IoT — Cascaded multi-granularity pruning with low-rank recovery enables extreme LLM compression for IIoT, guided by architectural independence.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.