RVPO: Risk-Sensitive Alignment via Variance Regularization

2026-05-08 · Source: Apple Machine Learning Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

Reward-Variance Policy Optimization (RVPO) is a new risk-sensitive framework designed to address constraint neglect in current critic-less Reinforcement Learning from Human Feedback (RLHF) methods. Existing methods, which aggregate multi-objective rewards using an arithmetic mean, can mask critical failures in objectives like safety or formatting when high success in another objective numerically offsets them. RVPO penalizes inter-reward variance during advantage aggregation, shifting the optimization objective from "maximize sum" to "maximize consistency." The framework utilizes a LogSumExp (SoftMin) operator, which acts as a smooth variance penalty. Evaluated on rubric-based medical and scientific reasoning with up to 17 LLM-judged reward signals (Qwen2.5-3B/7B/14B) and on tool-calling with rule-based constraints (Qwen2.5-1.5B/3B), RVPO improved overall scores on HealthBench (0.261 vs. 0.215 for GDPO at 14B, p < 0.001) and maintained competitive accuracy on GPQA-Diamond, mitigating constraint neglect across model scales.

Key takeaway

For research scientists developing multi-objective RLHF systems, RVPO offers a robust approach to prevent constraint neglect. By focusing on reward consistency rather than just sum maximization, you can ensure that critical constraints, such as safety or formatting, are not overlooked. Implementing RVPO's variance regularization can lead to more reliable and aligned models, particularly in complex domains like medical reasoning or tool-calling where diverse objectives must be met without sacrificing general capabilities.

Key insights

RVPO mitigates constraint neglect in multi-objective RLHF by penalizing inter-reward variance, enhancing consistency over sum maximization.

Principles

• Arithmetic mean aggregation can mask critical failures.
• Penalizing inter-reward variance improves consistency.
• LogSumExp acts as a smooth variance penalty.

Method

RVPO penalizes inter-reward variance during advantage aggregation using a LogSumExp (SoftMin) operator to shift the objective from maximizing sum to maximizing consistency.

In practice

• Apply RVPO for multi-objective LLM alignment.
• Use variance regularization to prevent constraint neglect.
• Improve reliability in safety-critical applications.

Topics

• RVPO
• Risk-Sensitive Alignment
• Constraint Neglect
• Multi-objective RLHF
• Variance Regularization

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization — BiasGRPO stabilizes LLM bias mitigation by normalizing rewards across completion groups, addressing high-variance subjective landscapes.
• KVPO: ODE-Native GRPO for Autoregressive Video Alignment via KV Semantic Exploration — KVPO aligns AR video generators by exploring semantic variations via KV cache routing and optimizing with an ODE-native velocity-field policy.
• Reducing Credit Assignment Variance via Counterfactual Reasoning Paths — Counterfactual trajectory comparison enables process-level credit assignment, reducing gradient variance in LLM reinforcement learning.
• MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following — MDP-GRPO stabilizes GRPO for multi-constraint instruction following by addressing z-score normalization pathologies with specific algorithmic enhancements.
• Rethinking the Divergence Regularization in LLM RL — DRPO improves LLM RL stability by replacing hard trust-region masks with a smooth, corrective divergence regularizer.
• Soft Sequence Policy Optimization — SSPO unifies sequence-level and soft policy optimization for stable, efficient off-policy LLM alignment.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Apple Machine Learning Research.