ReSET: Accurate Latency-Critical NVFP4 Reasoning via Step-Aware Temperature Scaling

2026-06-11 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

ReSET, a novel method, addresses the limitations of NVFP4 inference for Large Reasoning Models (LRMs) by improving both accuracy and latency. NVFP4 quantization typically degrades reasoning accuracy and existing kernels do not fully realize latency benefits in small-batch autoregressive decoding. ReSET analyzes how quantization increases incorrect sampling at low-entropy symbolic tokens and causes over-concentration in high-uncertainty reasoning steps. It proposes a reasoning-step entropy-based temperature-scaling method that adapts decoding temperature using both token-level and step-level entropy signals. Additionally, ReSET introduces a CUDA-core small-$M$ NVFP4 kernel for latency-critical autoregressive decoding. This approach improves NVFP4 reasoning accuracy by up to ~2 points and delivers up to 2.5x kernel-level speedup over NVFP4 vLLM, achieving approximately 2x end-to-end decoding speedup over BF16.

Key takeaway

For Machine Learning Engineers optimizing Large Reasoning Models with low-precision hardware, ReSET demonstrates that NVFP4 quantization can achieve significant speedups without severe accuracy degradation. You should investigate step-aware temperature scaling and specialized CUDA kernels to improve both inference latency and reasoning accuracy. Implementing these techniques could yield up to 2x end-to-end decoding speedup over BF16, making NVFP4 a more viable option for latency-critical applications.

Key insights

ReSET enhances NVFP4 LRM accuracy and latency via step-aware temperature scaling and a specialized CUDA kernel.

Principles

• Quantization increases incorrect sampling at low-entropy symbolic tokens.
• Quantization causes over-concentration on tokens in high-uncertainty reasoning steps.
• Decoding temperature can be adapted using token-level and step-level entropy.

Method

ReSET estimates step-level uncertainty online, adapting decoding temperature with token-level and step-level entropy signals, complemented by a CUDA-core small-$M$ NVFP4 kernel.

In practice

• Apply step-aware temperature scaling for quantized LRM inference.
• Utilize custom CUDA kernels for small-batch NVFP4 decoding.

Topics

• NVFP4
• Large Reasoning Models
• Quantization
• Temperature Scaling
• CUDA Kernels
• Autoregressive Decoding

Code references

• aiha-lab/ReSET

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• ReSET: Accurate Latency-Critical NVFP4 Reasoning via Step-Aware Temperature Scaling — Step-level uncertainty, not just token-level, is critical for accurate NVFP4 reasoning in Large Reasoning Models.
• ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization — ScaleSweep optimizes NVFP4 quantization by sweeping block scale candidates within derived theoretical bounds to minimize reconstruction error.
• QuickReduce FP4 Quantization and Benchmarking on MI355 — QuickReduce with FP4/INT4 quantization on MI355 significantly accelerates large-message all-reduce operations for LLM inference.
• Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillatio — Output-matching alone in LLM distillation can hide internal degradation; preserving internal geometry is crucial for accuracy.
• Dual-Dimensional Consistency: Balancing Budget and Quality in Adaptive Inference-Time Scaling — DDC optimizes LLM inference by adaptively balancing sampling budget and reasoning quality through a unified framework.
• Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery — Aggressive 2-bit quantization in LRMs causes specific generation pathologies, not just accuracy drops, which can be mitigated.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.