Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories

2026-07-12 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

Agentic-DPO is a novel, lightweight offline agent policy optimization method designed to enhance Large Language Model (LLM) agents beyond traditional supervised fine-tuning (SFT). While SFT merely imitates expert actions from trajectories, Agentic-DPO addresses the limitation of not learning to select optimal actions against potential errors. It transforms expert trajectories into state-conditioned preference supervision, sampling one-step actions at each expert state and contrasting plausible wrong actions with the expert action using a DPO-style objective. The method incorporates Policy-Preserving Augmentation (PPA) to maintain a fixed expert policy across multiple schemas, preventing policy-schema mixing. Agentic-DPO eliminates the need for online environment rollouts, reward models, or full-trajectory student exploration. Experiments across StableToolBench, tau-bench retail, and Mind2Web demonstrate consistent improvements, notably boosting tau-bench accuracy from 21.7% (SFT) to 41.4% for a 9B model, matching online GRPO performance without environment interaction during gradient steps. The code is available at https://github.com/Schuture/Agentic-DPO.

Key takeaway

For Machine Learning Engineers developing LLM agents, especially when online environment rollouts or reward model training are cost-prohibitive, you should consider Agentic-DPO. This method allows you to significantly improve agent performance and robustness beyond simple imitation learning by converting existing expert trajectories into state-conditioned preference supervision. Integrating Agentic-DPO can boost your agent's accuracy, as demonstrated by a 9B model's tau-bench accuracy rising from 21.7% to 41.4%, without incurring high online interaction costs.

Key insights

Agentic-DPO converts expert trajectories into state-level action preferences for efficient, offline LLM agent policy optimization, surpassing imitation learning.

Principles

• Offline preference learning improves agent robustness.
• State-conditioned preferences enhance action selection.
• Decouple policy and schema for stable learning.

Method

Agentic-DPO samples one-step actions at expert states, contrasting plausible wrong actions with expert actions using a DPO-style objective, aided by Policy-Preserving Augmentation.

In practice

• Apply DPO-style objectives to expert trajectories.
• Use Policy-Preserving Augmentation for schema invariance.
• Optimize LLM agents without costly online rollouts.

Topics

• Agentic-DPO
• LLM Agents
• Policy Optimization
• Preference Learning
• Offline RL
• Policy-Preserving Augmentation

Code references

• Schuture/Agentic-DPO

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Agentic RL: Environments, Trajectories, and the Training — Training multi-step LLM agents requires scoring full interaction trajectories using an LLM-as-a-judge within a GRPO framework.
• Understanding post-training of LLMs: DPO — DPO directly optimizes LLMs for human preferences without complex reward models or reinforcement learning loops.
• Learning When Not to Act: Mitigating Tool Abuse in Agentic Reinforcement Learning — EAPO enables agentic reinforcement learning models to selectively use tools, avoiding overuse without sacrificing performance.
• Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents — Semantic Consistency Policy Optimization (SCPO) resolves conflicting gradients in LLM agent reinforcement learning by assigning credit from successful sibling trajectories.
• StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning — Agentic RL requires aligning MDPs, credit assignment, and systems to the interaction step, not individual tokens.
• From GRPO to SAMPO: Solving Training Collapse in Agentic RL — SAMPO stabilizes agentic RL training by optimizing four key policy dimensions to prevent catastrophic gradient issues.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.