Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling

2026-07-02 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

A novel reinforcement learning algorithm, Correctness and Confidence Calibration Reinforcement Learning (C3RL), addresses the critical issue of poor confidence-accuracy calibration in large language models (LLMs) trained with RL. Current RL reward designs often prioritize correctness, leading to overconfident hallucinations when models are uncertain. C3RL integrates correctness, calibration, and dataset-informed reference accuracy rewards to mitigate this. Evaluated across 8 text and multimodal datasets, C3RL significantly enhances calibration without sacrificing accuracy, surpassing existing methods. Building on C3RL's well-calibrated verbalized confidence, the authors introduce Confidence-based Adaptive Test Time Scaling (CAS), an inference-time strategy that dynamically allocates computational resources. CAS outperforms majority voting on both in-domain and out-of-domain datasets, achieving up to a 12.33 times reduction in inference budget. This combined approach aims to deploy more reliable and resource-efficient LLMs.

Key takeaway

For Machine Learning Engineers deploying LLMs, addressing model overconfidence is crucial for reliability. Your teams should consider integrating confidence calibration into reinforcement learning pipelines, similar to C3RL, to prevent hallucinations and improve trustworthiness. Furthermore, leveraging well-calibrated confidence, as with CAS, allows you to dynamically adjust inference resources, potentially reducing your operational costs by over 12 times while maintaining or improving performance on diverse datasets.

Key insights

C3RL and CAS improve LLM reliability and efficiency by calibrating confidence and adaptively scaling inference.

Principles

• Reward designs should include confidence calibration.
• Calibrated confidence enables adaptive resource allocation.
• Overconfidence leads to hallucination.

Method

C3RL is an RL algorithm integrating correctness, calibration, and dataset-informed reference accuracy rewards. CAS is an inference strategy using C3RL's verbalized confidence for adaptive resource allocation.

In practice

• Implement C3RL for better LLM confidence.
• Use CAS to reduce LLM inference costs.
• Apply C3RL/CAS to multimodal datasets.

Topics

• Large Language Models
• Reinforcement Learning
• Model Calibration
• Inference Optimization
• Resource Efficiency
• Hallucination Prevention

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Making LLMs tell you how confident they really are through probe-targeted fine tuning.[R] — LLMs internally know their confidence but need fine-tuning to express it verbally, overcoming RLHF's bias.
• Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verification — TTRL-CoCoV improves LLM reasoning by confidence-conditioned verification, addressing pseudo-label errors and diversity collapse in label-free settings.
• Categories of Inference-Time Scaling for Improved LLM Reasoning — Inference-time scaling significantly improves LLM accuracy by applying more compute during generation, without altering model weights.
• An Empirical Study of Security Calibration in Large Language Models for Code — LLMs for code are systematically overconfident in security, especially in complex repository contexts, making their self-assessments unreliable.
• Adaptive Conformal Prediction for Improving Factuality of Generations by Large Language Models — Adaptive conformal prediction improves LLM factuality by calibrating uncertainty thresholds based on prompt characteristics, enhancing conditional coverage.
• Calibrating the Evaluator: Does Probability Calibration Mitigate Preference Coupling in LLM Agent Feedback Loops? — Probability calibration of LLM evaluator judgments significantly reduces preference coupling in agent feedback loops.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.