Why Adding More AI Agents Made Our System Slower

2026-07-23 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, medium

Summary

Planck experienced significant latency increases and timeout errors in their LLM agent system as the number of agents grew, despite using Python's asyncio for concurrent I/O. Initially, they suspected LLM providers, but diagnostics revealed their own code was the bottleneck. Their architecture, where a single service triggered multiple agents and sub-agents simultaneously, led to event loop lag. While I/O operations were concurrent, the cumulative CPU-bound tasks (e.g., JSON serialization/deserialization, data validation) after each LLM response overwhelmed the single event loop due to Python's Global Interpreter Lock (GIL). Optimizations like using orjson and adjusting aiohttp connection limits offered some improvement but did not resolve the core issue. The ultimate solution involved redesigning the system to distribute CPU work across multiple processes or workers, allowing for horizontal scaling and preventing a single point of contention.

Key takeaway

For AI Engineers scaling LLM agent systems in production, recognize that increasing concurrent I/O tasks will eventually bottleneck on cumulative CPU work within a single process. You should redesign your architecture to distribute CPU-bound operations across multiple workers or services. This prevents event loop lag and timeout errors, ensuring your system scales horizontally as you add more agents without degrading performance. Prioritize system design over micro-optimizations for long-term scalability.

Key insights

Asynchronous I/O removes waiting, but cumulative CPU-bound tasks after I/O operations become bottlenecks in single-process fan-out architectures.

Principles

• CPU work does not scale like I/O work.
• Asynchronous I/O only removes waiting, not computation.
• Distribute CPU work across processes for horizontal scaling.

Method

Redesign system architecture to distribute CPU-bound work across multiple processes, workers, or machines, rather than consolidating it in a single event loop.

In practice

• Use "orjson" for faster JSON serialization/deserialization.
• Adjust "aiohttp" connection limits for concurrent requests.
• Implement a router to fan out requests to multiple workers.

Topics

• LLM Agents
• Asynchronous Python
• System Scalability
• CPU Bottlenecks
• Distributed Systems
• Performance Optimization

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Autonomous Event-Driven Multi-Agent Orchestration for Enterprise AI at Scale — Enterprise AI multi-agent systems face significant performance degradation at scale due to agent discovery noise, not task complexity.
• The Compounding Latency Crisis of Multi-Step AI Workflows — Multi-step AI workflows suffer compounding latency from sequential LLM calls, requiring architectural optimization.
• Running the AI Factory: How Enterprises Operationalize AI Placement at Scale — Effective AI operationalization hinges on intelligent workload placement, especially for agentic AI, to optimize CPU/GPU utilization and control costs.
• Agents think in milliseconds, legacy infrastructure doesn't. LinkedIn, Walmart and Zendesk shared how they closed the gap at VB Transform 2026 — Legacy enterprise infrastructure, built for human speeds, is the primary bottleneck for AI agents in production, not the models themselves.
• Serving LLMs to Millions of Users The Production Playbook Part-4 — Scaling LLM systems requires architectural shifts beyond adding replicas, focusing on efficiency, cost, and resilience.
• The 100-agent benchmark: why enterprise AI scale stalls and how to fix it — Scaling agentic AI requires robust governance and unified platforms to overcome significant operational costs and production challenges.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.