CATPO: Critique-Augmented Tree Policy Optimization

2026-06-06 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

CATPO (Critique-Augmented Tree Policy Optimization) is a novel method designed to enhance the reasoning capabilities of large language models (LLMs) by optimizing reinforcement learning with verifiable rewards (RLVR). It specifically addresses computational waste in existing tree-based methods like TreeRPO. CATPO introduces a tree informativeness score, F(T), which combines leaf-outcome diversity with policy-reward decorrelation to identify valuable training trees without extra compute. For "dead-wrong" trees, it employs critique-guided healing, generating natural-language critiques and grafting refined continuations to recover training signal. An informativeness-weighted loss then scales gradient contributions. Experiments on Qwen2.5-Math-1.5B with the MATH dataset show CATPO achieves 37.5% macro accuracy across AIME24, MATH-500, OlympiadBench, and MinervaMath, improving over TreeRPO by 1.9% and GRPO by 4.8%.

Key takeaway

For machine learning engineers developing advanced LLM reasoning capabilities, especially in mathematical domains, CATPO offers a significant pathway to improve both accuracy and training efficiency. You should consider integrating its tree informativeness scoring and critique-guided healing mechanisms to enhance your model's learning signal and concentrate compute on the most impactful training data, potentially yielding substantial performance gains over current tree-based RL methods.

Key insights

CATPO optimizes LLM reasoning by focusing RL training on informative trees and healing failures with critiques.

Principles

• Tree informativeness can be quantified by leaf-outcome diversity and policy-reward decorrelation.
• Critique-guided healing recovers training signal from failed branches.
• Weighting gradient contributions by tree informativeness improves efficiency.

Method

Score trees with F(T) for informativeness, apply critique-guided healing to dead-wrong trees, and use an informativeness-weighted loss for gradient updates.

In practice

• Implement F(T) to identify valuable training trees.
• Generate natural-language critiques for shallow failure points.
• Graft refined continuations to failed branches.

Topics

• Large Language Models
• Reinforcement Learning
• Policy Optimization
• Tree-based Methods
• Critique-guided Learning
• Mathematical Reasoning

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards — CIPO enhances LLM reasoning by converting failed trajectories into explicit, directional correction signals for policy optimization.
• ATPO: Adaptive Tree Policy Optimization for Multi-Turn Medical Dialogue — ATPO uses uncertainty-aware adaptive tree search and efficient execution to optimize multi-turn medical dialogue LLMs.
• The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning — Adaptive policy optimization, using question difficulty and model confidence, significantly improves LLM reasoning efficiency and accuracy over uniform training.
• EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training — Adaptive critic utilization in LLM post-training can reduce advantage variance more effectively than fixed baselines.
• Reinforcement Learning Improves LLM Accuracy and Reasoning in Disease Classification from Radiology Reports — Reinforcement learning with GRPO enhances LLM accuracy and reasoning in medical text classification without explicit reasoning supervision.
• Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization — CW-GRPO improves LLM search agents by using an LLM judge for fine-grained, per-round credit assignment.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.