Vector Policy Optimization: Training for Diversity Improves Test-Time Search

2026-05-21 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

Vector Policy Optimization (VPO) is a novel reinforcement learning algorithm designed to address the limitations of standard language model post-training, which often optimizes for a single scalar reward and produces low-entropy response distributions. This lack of diversity hinders LLMs in inference-scaling search procedures like AlphaEvolve, which require varied rollouts. VPO explicitly trains policies to anticipate diverse downstream reward functions and generate diverse solutions by leveraging the often vector-valued nature of rewards, such as per-test-case correctness in code generation or multiple user personas. Functioning as a drop-in replacement for the GRPO advantage estimator, VPO trains LLMs to output solution sets where individual solutions specialize in different trade-offs within the vector reward space. Across four distinct tasks, VPO consistently matches or surpasses strong scalar RL baselines on test-time search metrics like pass@k and best@k, with performance advantages increasing with larger search budgets. Notably, VPO models enable the solution of evolutionary search problems that GRPO models cannot tackle.

Key takeaway

For Machine Learning Engineers developing LLMs for complex, multi-objective tasks, consider adopting Vector Policy Optimization (VPO) in your post-training pipeline. This approach directly addresses the limitations of scalar reward optimization by explicitly training your models for output diversity, which is crucial for robust inference-scaling search procedures. Implementing VPO can significantly improve your LLM's performance on metrics like pass@k and best@k, especially as search budgets increase, allowing you to solve problems that traditional methods cannot achieve.

Key insights

Vector Policy Optimization trains LLMs for diversity, significantly improving test-time search performance.

Principles

• Scalar rewards limit LLM output diversity.
• Vector rewards enable diverse solution specialization.
• Diversity improves inference-time search efficacy.

Method

VPO is an RL algorithm replacing the GRPO advantage estimator, training LLMs to output solution sets where individual solutions specialize to different trade-offs in vector reward space.

In practice

• Implement VPO for LLM post-training.
• Define vector rewards for diverse objectives.
• Enhance pass@k and best@k search metrics.

Topics

• Vector Policy Optimization
• Reinforcement Learning
• Language Models
• Diversity Training
• Test-Time Search
• Vector Rewards

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Vector Policy Optimization: Training for Diversity Improves Test-Time Search — Training LLMs for reward diversity significantly improves performance in inference-time search procedures.
• Week Ending 5.17.2026 — Vector Policy Optimization trains LLMs for diverse, specialized outputs, improving performance in inference-time search.
• EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training — Adaptive critic utilization in LLM post-training can reduce advantage variance more effectively than fixed baselines.
• VIMPO: Value-Implicit Policy Optimization for LLMs — VIMPO offers critic-free policy optimization for LLMs by deriving a policy-implied value function, improving credit assignment and performance on verifiable reward tasks.
• Rethinking the Divergence Regularization in LLM RL — DRPO improves LLM RL stability by replacing hard trust-region masks with a smooth, corrective divergence regularizer.
• Diverse reasoning traces teach LLMs to make better decisions — Training LLMs with diverse reasoning traces and explicit mode selection tokens improves accuracy and prevents mode collapse.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.