Learning to Construct Practical Agentic Systems
Summary
Principled approaches for designing and optimizing practical agentic systems are introduced, focusing on production concerns like simplicity, controllability, and predictable inference costs. The core is an agent framework that enforces modularity by defining "pseudo-tools," which recursively invoke LLMs on restricted contexts. Through hand-engineering agents for diverse tasks, the study demonstrates that fixed workflows constructed within this framework are generally cheaper and more accurate compared to dynamically-planned workflows. The paper then presents novel learning methods specifically for these agentic components, including pseudo-tools and fixed workflows. These learned methods consistently outperform their hand-engineered counterparts. The framework's inherent modularity also enables multi-objective optimization to jointly enhance both cost efficiency and response quality, allowing for the blending of results from multiple learning systems.
Key takeaway
For MLOps Engineers building agentic LLM systems, prioritize designing with modularity and fixed workflows. Your systems will likely be cheaper and more accurate than dynamically-planned alternatives. Consider implementing the proposed learning methods to optimize pseudo-tools and workflows, as these demonstrably outperform hand-engineered solutions. This approach allows for multi-objective optimization, enabling you to balance inference costs with response quality effectively.
Key insights
Modular agentic systems with learned fixed workflows offer superior cost and accuracy over dynamic planning.
Principles
- Enforce modularity via "pseudo-tools" for restricted LLM calls.
- Fixed workflows generally offer better cost and accuracy than dynamic planning.
- Learning methods can outperform hand-engineered agent components.
Method
The framework defines "pseudo-tools" for modularity, allowing recursive LLM calls on restricted contexts. It then applies novel learning methods to optimize these pseudo-tools and fixed workflows, followed by multi-objective optimization for cost and quality.
In practice
- Design agents with modular "pseudo-tools."
- Favor fixed workflows for cost-effective accuracy.
- Optimize agent components using learning methods.
Topics
- Agentic Systems
- LLM Agents
- Modular Architectures
- Workflow Optimization
- Multi-objective Optimization
- Machine Learning Methods
Best for: AI Architect, AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, MLOps Engineer
Related on AIssential
Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.