Learning the Supports for Categorical Critic in Reinforcement Learning

2026-07-02 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

This work introduces an approach to dynamically learn the lower and upper bounds of the support interval for categorical critics in reinforcement learning (RL). Conventionally, value functions in actor-critic RL are trained via mean squared error (MSE) regression, while distributional RL models return distributions. The Gaussian Histogram Loss (HL-Gauss) reframes value estimation as classification, encoding scalar Bellman targets as Gaussian-smoothed categorical targets. A key challenge with HL-Gauss is the requirement for a pre-defined fixed support interval, complicated by the non-stationary and stochastic nature of RL target values. The proposed method derives an objective that jointly learns these bounds and the categorical representation, forming an upper bound on the mean-squared Bellman error. Theoretical analysis indicates this bound is tighter than non-learned HL-Gauss supports. Empirically, the objective enables stable support adaptation, matching HL-Gauss on most continuous-control tasks and improving on a subset, without needing pre-specified intervals.

Key takeaway

For Machine Learning Engineers developing actor-critic reinforcement learning systems, you should consider implementing dynamically learned support bounds for categorical critics. This approach eliminates the need to pre-define fixed support intervals, a common challenge with methods like Gaussian Histogram Loss, leading to more stable adaptation and potentially improved performance on continuous-control tasks. Integrating this method can simplify model configuration and enhance robustness in your RL deployments.

Key insights

Dynamically learning support bounds for categorical critics in reinforcement learning overcomes fixed-interval limitations, improving stability.

Principles

• Dynamically adapting support intervals enhances RL stability.
• Jointly learning bounds and categorical representations tightens error.
• Reframing value estimation as classification is effective.

Method

Derive an objective to jointly learn lower and upper support bounds while simultaneously learning the categorical representation of scalar values.

In practice

• Apply to continuous-control tasks for performance gains.
• Automate support interval definition in distributional RL.

Topics

• Reinforcement Learning
• Distributional RL
• Categorical Critic
• Value Functions
• Gaussian Histogram Loss
• Continuous Control

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Representation Learning Enables Scalable Multitask Deep Reinforcement Learning — Scalable multitask reinforcement learning is primarily driven by representation learning, not complex model-based planning.
• CATPO: Critique-Augmented Tree Policy Optimization — CATPO optimizes LLM reasoning by focusing RL training on informative trees and healing failures with critiques.
• Optimistic Policy Learning under Pessimistic Adversaries with Regret and Violation Guarantees — RHC-UCRL enables safe, optimal reinforcement learning against explicit adversarial policies with theoretical guarantees.
• On Distributional Reinforcement Learning in Chaotic Dynamical Systems — Chaotic systems challenge RL, but distributional RL offers better conditioned learning by utilizing the $1$-Wasserstein metric's stability.
• Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning — Continual RL benefits from maintaining diverse policy neighborhoods, not just single-model preservation.
• ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Information — ASymPO stabilizes asynchronous LLM post-training by normalizing token loss with current policy probabilities, eliminating the need for behavior information.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.