Hint-Guided Diversified Policy Optimization for LLM Reasoning

2026-07-27 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

Hint-Guided Diversified Policy Optimization (HDPO) is a novel framework designed to enhance Large Language Model (LLM) reasoning by explicitly guiding the model to explore diverse solution paths. HDPO operates through a "propose-select-think" trajectory, where the LLM first generates multiple candidate solution outlines (hints) and then selects the most reliable one for detailed reasoning. This two-stage process involves a Cold Start for Structured Reasoning using supervised fine-tuning with data distilled from an advanced LLM, followed by Hint-Guided Diversified Reinforcement Learning. The RL phase incorporates unique rewards for format, accuracy, diversity (scheduled with a sine function and gated by correctness), and reliability (proxied by token entropy). Experiments on Qwen3-4B, Qwen2.5-Math-7B, and DeepSeek-R1-Distill-Llama-8B across nine benchmarks, including OlympiadBench and MMLU-Pro, demonstrate that HDPO significantly boosts LLM reasoning, improving accuracy by 8.24% and 7.58% over GRPO on Qwen3-4B and Qwen2.5-Math-7B respectively, while maintaining low inference latency.

Key takeaway

For Machine Learning Engineers developing advanced LLM reasoning systems, you should consider integrating Hint-Guided Diversified Policy Optimization (HDPO) to improve solution robustness and accuracy. Adopt its "propose-select-think" framework and incorporate explicit diversity and reliability rewards. Your models will then explore broader solution spaces and select more dependable reasoning paths. This outperforms traditional accuracy-centric RL methods with minimal inference latency increase, offering a scalable path for continuous model optimization.

Key insights

Diversifying solution exploration and explicitly incentivizing reliability significantly improves LLM reasoning performance.

Principles

• LLM reasoning benefits from multi-path exploration.
• Reward diversity and reliability in RL for robust solutions.
• Scheduled rewards can balance early accuracy with later exploration.

Method

HDPO uses a two-stage "propose-select-think" trajectory: SFT for structured reasoning, then RL with format, accuracy, scheduled diversity, and entropy-based reliability rewards.

In practice

• Implement a "propose-select-think" prompt structure.
• Use cosine similarity for diversity reward calculation.
• Employ token entropy as a proxy for solution reliability.

Topics

• Large Language Models
• Reinforcement Learning
• Policy Optimization
• Reasoning Benchmarks
• Diversity Rewards
• Reliability Estimation
• "Propose-Select-Think"

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning — Adaptive policy optimization, using question difficulty and model confidence, significantly improves LLM reasoning efficiency and accuracy over uniform training.
• Beyond Mode Collapse: Distribution Matching for Diverse Reasoning — Distribution matching prevents mode collapse in on-policy RL by promoting diverse exploration, improving reasoning capabilities.
• Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning — Strategy-Guided Policy Optimization distills reasoning by transferring reusable problem-solving strategies, not just specific solution steps.
• H$^2$SD: Hybrid Hindsight Self-Distillation — H$^2$SD improves LLM reasoning by adaptively applying self-distillation based on trajectory correctness.
• Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards — CIPO enhances LLM reasoning by converting failed trajectories into explicit, directional correction signals for policy optimization.
• Diverse reasoning traces teach LLMs to make better decisions — Training LLMs with diverse reasoning traces and explicit mode selection tokens improves accuracy and prevents mode collapse.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.