Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards

2026-05-14 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

Correction-Oriented Policy Optimization (CIPO) is a novel extension to Reinforcement Learning with Verifiable Rewards (RLVR) designed to enhance large language models' reasoning capabilities. CIPO addresses the limitations of sparse binary rewards and weak credit assignment in traditional RLVR by converting failed on-policy trajectories into correction-oriented supervision. This method allows models to learn from their own errors without external signals, jointly optimizing these correction samples with the standard RLVR objective. Extensive experiments across 11 benchmarks in mathematical reasoning and code generation demonstrate that CIPO consistently and significantly outperforms strong baselines in both reasoning and correction performance, yielding stronger pass@K gains and improving intrinsic reasoning capacity.

Key takeaway

For AI engineers developing large language models, CIPO offers a robust method to improve reasoning and self-correction. By integrating CIPO, your models can learn more effectively from their own mistakes, leading to significant performance gains in complex tasks like mathematical reasoning and code generation. Consider implementing CIPO to enhance intrinsic reasoning capacity and achieve stronger pass@K results.

Key insights

CIPO enhances RLVR by transforming failed trajectories into self-correction supervision, improving LLM reasoning.

Principles

• Learn from internal failures
• Convert errors into supervision

Method

CIPO jointly optimizes correction samples derived from a model's own failed attempts with the standard RLVR objective, creating self-correction supervision without external signals.

In practice

• Apply CIPO to LLM training
• Improve mathematical reasoning
• Enhance code generation

Topics

• Reinforcement Learning with Verifiable Rewards
• Correction-Oriented Policy Optimization
• Large Language Models
• Mathematical Reasoning
• Code Generation

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards — CIPO enhances LLM reasoning by converting failed trajectories into explicit, directional correction signals for policy optimization.
• ESPO: Early-Stopping Proximal Policy Optimization — ESPO improves LLM reinforcement learning by early-stopping failed trajectories based on real-time regret estimation.
• CAST: Non-Privileged Clipped Asymmetric Self-Teaching with Advantage Flipping for GRPO — CAST enhances GRPO-style RLVR by using an answer-free self-teacher and bidirectional advantage flipping for dense, aligned token-level feedback.
• Overcoming reward signal challenges: Verifiable rewards-based reinforcement learning with GRPO on SageMaker AI — RLVR with GRPO and few-shot examples significantly improves LLM accuracy on verifiable tasks.
• GraphPO: Graph-based Policy Optimization for Reasoning Models — GraphPO uses a graph-based RL framework to merge semantically similar reasoning states, reducing redundancy and improving exploration efficiency.
• Momentum for Reasoning: Dense Intrinsic Signals in Policy Optimization — ISPO uses dense intrinsic signals to overcome common failure modes in RLVR for LLM reasoning tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.