VIMPO: Value-Implicit Policy Optimization for LLMs

2026-06-18 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

VIMPO introduces a novel critic-free policy optimization method designed to enhance the reasoning abilities of large language models (LLMs) using reinforcement learning with verifiable rewards. This approach addresses the inherent trade-off between method simplicity and effective credit assignment found in existing techniques like GRPO, which assigns trajectory-level advantages, and actor-critic methods, which require a potentially unstable learned value function. VIMPO derives a policy-implied value function from KL-regularized reinforcement learning optimality conditions, enabling a simple value loss that incorporates outcome-level verifiable rewards without needing a separate critic. It also provides a critic-free actor advantage, allowing distinct reward incorporation and PPO-style policy improvement. VIMPO demonstrates improved performance over GRPO across mathematical RLVR benchmarks, including MATH-500, AIME 2024, AIME 2025, and OlympiadBench, showing larger gains on competition-style tasks and maintaining consistency even with noisy rewards.

Key takeaway

For Machine Learning Engineers fine-tuning LLMs for complex reasoning tasks, VIMPO presents a compelling alternative to traditional reinforcement learning approaches. If your projects demand improved credit assignment with verifiable rewards but struggle with actor-critic training instability or GRPO's coarse advantages, you should evaluate VIMPO. Its critic-free, policy-implied value function offers a simpler, more stable path to enhanced performance on benchmarks like mathematical RLVR, even under noisy reward conditions.

Key insights

VIMPO offers critic-free policy optimization for LLMs by deriving a policy-implied value function, improving credit assignment and performance on verifiable reward tasks.

Principles

• Policy-implied values can replace learned critics.
• KL-regularized RL optimality yields value recurrence.
• Separating reward incorporation from policy updates.

Method

VIMPO derives a policy-implied value function from KL-regularized RL optimality, using a value recurrence based on policy-reference log-ratios and a terminal condition, then applies a PPO-style actor update.

In practice

• Improve LLM reasoning on math benchmarks.
• Enhance credit assignment in RLVR tasks.
• Apply critic-free methods for stable training.

Topics

• Large Language Models
• Reinforcement Learning
• Policy Optimization
• Critic-Free RL
• Credit Assignment
• Mathematical Reasoning

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Value-Gradient Hypothesis of RL for LLMs — Critic-free RL for LLMs works because the actor's backward pass implicitly propagates a value-gradient signal, approximated via attention.
• The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning — Adaptive policy optimization, using question difficulty and model confidence, significantly improves LLM reasoning efficiency and accuracy over uniform training.
• Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards — SC-GRPO improves RLVR credit assignment by using self-conditioned KL divergence to weight gradients, outperforming uniform methods.
• Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards — Augmenting PPO with multi-step likelihood ratios provides a principled way to control bias-variance in RLVR.
• RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning — POPO improves LLM reasoning by efficiently leveraging effective off-policy data in RLVR, reducing computational overhead.
• Vector Policy Optimization: Training for Diversity Improves Test-Time Search — Vector Policy Optimization trains LLMs for diversity, significantly improving test-time search performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.