ReCal: Reward Calibration for RL-based LLM Routing

2026-06-10 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

ReCal, a novel Reward Calibration framework, addresses challenges in reinforcement learning (RL)-based large language model (LLM) routing. Current RL methods for LLM routing, which dynamically select models and reasoning strategies, struggle with providing comparable learning signals across heterogeneous tasks. This is due to aggregating multiple objectives like correctness and format behavior into a single scalar reward, leading to ambiguous credit assignment and conflicting optimization signals. Additionally, reward variability across instances introduces optimization bias, favoring trivial samples. ReCal tackles these issues by introducing a hierarchical reward decomposition mechanism with component-wise advantage estimation. It also employs a distribution-aware optimization strategy, incorporating variance-aware reweighting and per-dataset normalization. Experiments across seven datasets demonstrate that ReCal consistently enhances routing performance and training stability compared to existing baselines.

Key takeaway

For Machine Learning Engineers optimizing LLM routing, if you are struggling with unstable training or ambiguous reward signals, consider implementing ReCal's reward calibration techniques. Your systems can achieve more consistent performance and stable training by decomposing rewards hierarchically and applying distribution-aware optimization. This approach directly addresses common pitfalls of scalar reward aggregation and reward variability, leading to improved routing quality.

Key insights

ReCal improves RL-based LLM routing by calibrating rewards through hierarchical decomposition and distribution-aware optimization, enhancing learning signals.

Principles

• Scalar reward aggregation creates ambiguous signals.
• Reward variability biases optimization towards trivial samples.
• Hierarchical reward decomposition clarifies credit assignment.

Method

ReCal employs hierarchical reward decomposition with component-wise advantage estimation. It further uses a distribution-aware optimization strategy, applying variance-aware reweighting and per-dataset normalization to calibrate rewards.

In practice

• Decompose complex rewards hierarchically.
• Reweight samples based on reward variance.
• Normalize rewards per dataset for consistency.

Topics

• LLM Routing
• Reinforcement Learning
• Reward Calibration
• Hierarchical Reward Decomposition
• Distribution-Aware Optimization
• Large Language Models

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• ReCal: Reward Calibration for RL-based LLM Routing — ReCal calibrates RL-based LLM routing rewards by disentangling objectives and normalizing distributions for clearer, more stable policy learning.
• Beyond Distribution Sharpening: The Importance of Task Rewards — Task-reward optimization in LLM fine-tuning offers stable, superior performance over distribution sharpening, especially for complex tasks.
• LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking — LLMs trained with RLVR can exploit imperfect verifiers through reward hacking, bypassing genuine rule induction.
• ConSteer-RL: Steering Reasoning Capabilities in Large Language Models via Confidence-Aware Reinforcement Learning — ConSteer-RL improves LLM reasoning by integrating token-level confidence into RLVR, penalizing overconfident errors and reinforcing confident, correct reasoning.
• Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces — RL for LLM multi-agent systems requires optimizing orchestration through temporal interaction graphs.
• Value-Gradient Hypothesis of RL for LLMs — Critic-free RL for LLMs works because the actor's backward pass implicitly propagates a value-gradient signal, approximated via attention.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.