Vector Policy Optimization: Training for Diversity Improves Test-Time Search

2026-05-22 · Source: cs.NE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

Vector Policy Optimization (VPO) is a novel reinforcement learning algorithm designed to enhance language model performance in inference-scaling search procedures. Traditional LLM post-training optimizes for a single scalar reward, resulting in low-diversity outputs that hinder advanced search methods like AlphaEvolve. VPO addresses this by explicitly training policies to generate diverse solutions that anticipate varied downstream reward functions. It operates as a drop-in replacement for the GRPO advantage estimator, leveraging vector-valued rewards (e.g., per-test-case correctness) and stochastic reward scalarizations. Across four tasks—Maze, MuSiQue, EUREQA, and ToolRL—VPO consistently matches or surpasses scalar RL baselines on best@k metrics, with the performance gap increasing with search budget. Notably, VPO-trained Qwen2.5-Coder-7B-Instruct models on LiveCodeBench unlocked problems that GRPO could not solve, demonstrating its effectiveness in complex evolutionary search scenarios.

Key takeaway

For machine learning engineers developing LLM-powered systems that incorporate test-time search, you should consider adopting Vector Policy Optimization (VPO) for post-training. This approach, which prioritizes generating diverse, high-quality candidate solutions over single optimal responses, will significantly improve your system's ability to leverage search budgets effectively. By enabling your models to cover a broader Pareto front of solutions, VPO can unlock performance on complex problems that traditional scalar reward optimization methods cannot address.

Key insights

Training LLMs for reward diversity significantly improves performance in inference-time search procedures.

Principles

• Separate exploration (training) from exploitation (search.
• Vector-valued rewards enable natural diversity optimization.
• Stochastic reward scalarization incentivizes solution specialization.

Method

VPO combines multi-answer generation (e.g., m=3 candidates per rollout) with stochastic reward scalarization, where weights w~Dir(α) are sampled, and the model is rewarded for producing sets that span the Pareto frontier.

In practice

• Use VPO for LLMs integrated into search-augmented systems.
• Apply VPO where task rewards naturally decompose into multiple components.

Topics

• Vector Policy Optimization
• Reinforcement Learning
• Language Models
• Test-Time Search
• Reward Diversity
• Multi-objective Optimization

Code references

• algorithmicsuperintelligence/openevolve

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Vector Policy Optimization: Training for Diversity Improves Test-Time Search — Vector Policy Optimization trains LLMs for diversity, significantly improving test-time search performance.
• Week Ending 5.17.2026 — Vector Policy Optimization trains LLMs for diverse, specialized outputs, improving performance in inference-time search.
• EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training — Adaptive critic utilization in LLM post-training can reduce advantage variance more effectively than fixed baselines.
• VIMPO: Value-Implicit Policy Optimization for LLMs — VIMPO offers critic-free policy optimization for LLMs by deriving a policy-implied value function, improving credit assignment and performance on verifiable reward tasks.
• OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation — OrderGrad enables direct optimization of reward distribution properties beyond the mean using rank-weighted gradient estimators.
• Beyond Mode Collapse: Distribution Matching for Diverse Reasoning — Distribution matching prevents mode collapse in on-policy RL by promoting diverse exploration, improving reasoning capabilities.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.NE updates on arXiv.org.