In-the-Flow Agentic System Optimization for Effective Planning and Tool Use

2026-07-23 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

AgentFlow is a novel trainable, in-the-flow agentic framework designed to optimize large language models for effective planning and tool use in complex, multi-turn interactions. It coordinates a planner, executor, verifier, and generator via an evolving memory, directly optimizing the planner within the live multi-turn loop. The system employs Flow-based Group Refined Policy Optimization (Flow-GRPO) to address long-horizon, sparse-reward credit assignment by broadcasting a single trajectory-level outcome to every turn and stabilizing learning with group-normalized advantages. Using a 7B-scale backbone, AgentFlow achieved average accuracy gains of 14.9% on search, 14.0% on agentic, 14.5% on mathematical, and 4.1% on scientific tasks, even surpassing GPT-4o.

Key takeaway

For AI Architects designing advanced LLM applications requiring multi-turn reasoning and diverse tool use, you should evaluate AgentFlow's "in-the-flow" agentic system. Its on-policy optimization via Flow-GRPO significantly enhances planning and tool-calling reliability, outperforming larger monolithic models. This approach offers a robust framework for tackling long-horizon tasks, demonstrating superior performance and positive scaling with increased reasoning turns.

Key insights

AgentFlow optimizes LLM agent planning on-policy in multi-turn interactions using Flow-GRPO for robust, long-horizon task execution.

Principles

• Decompose complex tasks into specialized modules.
• Optimize planners directly within live interaction loops.
• Broadcast global outcomes for multi-turn credit assignment.

Method

Flow-GRPO converts multi-turn RL into single-turn policy updates by broadcasting a final outcome reward to every turn and using group-normalized advantages for stable on-policy training.

In practice

• Use Qwen2.5-7B-Instruct as a module backbone.
• Employ GPT-4o as an LLM-based reward judge.
• Increase inference turn budgets for deeper reasoning.

Topics

• AgentFlow
• Agentic Systems
• Reinforcement Learning
• LLM Tool Use
• On-policy Optimization
• Multi-turn Reasoning

Code references

• lupantech/AgentFlow

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Learning to Construct Practical Agentic Systems — Modular agentic systems with learned fixed workflows offer superior cost and accuracy over dynamic planning.
• Tools as Continuous Flow for Evolving Agentic Reasoning — FlowAgent reconceptualizes LLM tool chaining as continuous trajectory generation for robust, generalized reasoning.
• Reinforced Agent: Inference-Time Feedback for Tool-Calling Agents — Proactive, in-execution review by a specialized agent improves tool-calling agent reliability and performance.
• Ralph Wiggum (and why Claude Code's implementation isn't it) with Geoffrey Huntley and Dexter Horthy — Effective LLM agent design requires deterministic context management and human oversight, not autonomous "in-loop" operation.
• From Atomic Actions to Standard Operating Procedures: Iterative Tool Optimization for Self-Evolving LLM Agents — LLM agents can self-evolve by iteratively optimizing toolsets through SOP synthesis and systematic management.
• "My agentic engineering workflow" | George Pickett (MTS @ Parallel) — Clarifying intent and creating durable artifacts are crucial for effective, long-running AI agent workflows.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.