Performance engineering in agentic AI systems

2026-07-06 · Source: Thoughtworks Insights · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, long

Summary

Enterprise AI is evolving towards orchestrated multi-agent systems, but naive designs face challenges like accumulating latency, escalating token costs, and reliability issues. This article, published July 06, 2026, by Divye Singh and Limansha Safreen Shaik, details optimizations to transition agentic AI from prototypes to production. Key strategies include right-sizing models for specific tasks, implementing prompt and API response caching (both key-value and semantic), and leveraging async parallelization for independent workflow steps. It also advocates for partial deterministic flows to reduce LLM calls, dynamic context engineering to minimize irrelevant input, and structuring prompts for optimal KV cache utilization. The authors also suggest using semantic similarity over LLM classification for stable label spaces, noting its cost, latency, and debuggability advantages.

Key takeaway

For AI Architects designing enterprise-scale agentic systems, prioritize performance engineering from the outset to avoid escalating costs and latency. You should implement granular observability, right-size models for specific task complexities, and strategically apply caching and parallelization. Dynamically manage context and structure prompts for KV cache efficiency. Consider semantic similarity for stable classifications to improve debuggability and reduce per-call costs, ensuring your systems are operable and cost-effective under production load.

Key insights

Optimizing agentic AI systems requires strategic model selection, intelligent caching, parallelization, and precise context management to reduce cost and latency.

Principles

• Measure latency, token consumption, and failure rates at every node.
• Match task complexity to the appropriate model tier.
• Treat context as a dynamic selection problem, not static inclusion.

Method

Implement observability for latency, token consumption, and failure rates. Map model tiers to task complexity. Use fan-out/fan-in for parallel execution. Structure prompts with stable prefixes and dynamically selected context for KV cache hits.

In practice

• Assign small models for low-complexity tasks like intent routing.
• Cache semantically similar queries to avoid re-running workflows.
• Parallelize independent workflow steps using fan-out/fan-in.

Topics

• Agentic AI Systems
• LLM Performance
• Model Right-Sizing
• Semantic Caching
• Prompt Engineering
• Workflow Parallelization

Best for: AI Engineer, MLOps Engineer, AI Architect

Related on AIssential

• Quoting Thariq Shihipar — Prompt caching is essential for cost-effective, low-latency agentic AI products.
• Context Engineering for AI Agents: Part 2 — Effective AI agent performance hinges on precise context management, minimizing irrelevant information and optimizing tool access.
• I Tried to Design Google’s Enterprise Data Agent in a Mock Interview — Here’s Where My 5-Second… — Achieving sub-5-second latency for enterprise AI agents requires architectural shifts beyond simple LLM orchestration.
• not much happened today — AI model development is rapidly advancing across open and closed source, with a focus on cost-efficiency, agentic capabilities, and robust infrastructure.
• Graph-Validated Memory Architecture: Enhancing Contextual Accuracy and Retrieval Speed in Agentic… — The Graph-Validated Memory Architecture resolves the latency-accuracy trade-off in agentic systems by combining fast semantic caching with knowledge graph validation.
• Agentic AI: How to Save on Tokens — Optimizing LLM agent costs requires strategic token reuse, context minimization, intelligent model routing, and diligent context cleaning.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Thoughtworks Insights.