STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training

2026-07-06 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

Selective Trajectory-Aware Policy Optimization (STAPO) is a new hierarchical group-based Reinforcement Learning framework designed to train Large Language Model (LLM) agents for long-horizon tasks. It specifically tackles "trajectory neglect," a common issue where agents lose focus due to sparse and delayed rewards. Unlike prior methods that use Shannon-entropy for step-level supervision, STAPO introduces "normalized entropy" to more reliably measure confidence deviations and identify low-quality actions. This framework then optimizes these outlier steps using a joint mechanism of trajectory-aware reward and trajectory-independent penalty. Extensive experiments on ALFWorld, WebShop, and Search-Augmented QA demonstrate that STAPO achieves leading performance while substantially alleviating trajectory neglect, validating its effectiveness and robustness for agentic tasks.

Key takeaway

For Machine Learning Engineers training LLM agents on long-horizon tasks, STAPO offers a robust solution to combat trajectory neglect. You should consider integrating normalized entropy for more accurate step-level supervision, moving beyond traditional Shannon-entropy. Implementing STAPO's joint trajectory-aware reward and trajectory-independent penalty mechanism can significantly enhance agent stability and performance, especially on benchmarks like ALFWorld or WebShop.

Key insights

STAPO uses normalized entropy and a dual optimization mechanism to mitigate trajectory neglect in LLM agents, achieving leading performance.

Principles

• Trajectory neglect hinders LLM agent performance.
• Normalized entropy improves confidence estimation.
• Dual optimization enhances trajectory awareness.

Method

STAPO employs normalized entropy to identify outlier steps, then optimizes them using a joint trajectory-aware reward and trajectory-independent penalty within a hierarchical group-based RL framework.

In practice

• Apply normalized entropy for reliable decision reliability.
• Use dual reward/penalty for agent optimization.
• Test on ALFWorld, WebShop, Search-Augmented QA.

Topics

• LLM Agents
• Reinforcement Learning
• Trajectory Optimization
• Normalized Entropy
• Policy Optimization
• ALFWorld
• WebShop

Best for: Research Scientist, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learning — TAO-RL stabilizes agentic reinforcement learning by filtering tool-use trajectories and guiding exploration with entropy for efficient LLM policy optimization.
• LLMZero: Discovering Adaptive Training Strategies for RL Post-Training via LLM Agents — LLMZero uses LLM agents and tree search to adaptively optimize RL post-training, outperforming fixed schedules.
• Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning — Strategy-Guided Policy Optimization distills reasoning by transferring reusable problem-solving strategies, not just specific solution steps.
• From GRPO to SAMPO: Solving Training Collapse in Agentic RL — SAMPO stabilizes agentic RL training by optimizing four key policy dimensions to prevent catastrophic gradient issues.
• GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents — LLM-based agents can autonomously optimize Off-Policy Evaluation code, significantly improving performance and reliability in data-driven systems.
• What Makes an LLM a Good Optimizer? A Trajectory Analysis of LLM-Guided Evolutionary Search — Strong LLM optimizers act as local refiners, making incremental improvements within localized semantic search spaces.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.