Most AI Work Can Wait

2026-07-01 · Source: Tomasz Tunguz · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Intermediate, short

Summary

This article advocates for prioritizing AI agent routing architecture over initial model selection to significantly reduce operational costs and improve efficiency. It proposes a three-layered routing system: a skill classifier identifies the task, a router decides which model tier (local or async) executes it based on factors like complexity and context size, and a model selector picks the cheapest model within that tier. This approach enables 70-80% of traffic to run on local models or async batch inference, potentially cutting AI spend by 90%+, as exemplified by Coinbase's 50% cost reduction despite increased token usage. The system incorporates synchronous failure-mode signals and nightly closed-loop feedback to continuously optimize routing decisions, catching known-hard tasks and discovering new failure modes.

Key takeaway

For AI Architects and MLOps Engineers designing agent systems, prioritize building a robust routing layer before selecting specific models. This strategy allows you to direct 70-80% of requests to cheaper local or asynchronous models, significantly reducing inference costs by up to 90%+. Implement a multi-layered router with continuous feedback loops to dynamically optimize model usage and ensure efficient resource allocation, freeing up budget for more complex, real-time tasks.

Key insights

Prioritize AI agent routing architecture over model choice to optimize cost and performance.

Principles

• Model choice is the last decision.
• Router is a scheduling problem.
• Queueing enables significant cost savings.

Method

Implement a three-layer routing system: skill classifier, router, and model selector. Integrate synchronous failure-mode signals and nightly closed-loop feedback for continuous optimization.

In practice

• Route 70-80% of traffic to local models.
• Utilize async batch inference for non-real-time tasks.
• A/B test models against the same operation.

Topics

• AI Agent Architecture
• Model Routing
• Inference Cost Optimization
• Asynchronous Inference
• Skill Distillation
• MLOps

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Engineer, MLOps Engineer, AI Architect

Related on AIssential

• How Small AI Models Can Cut Your AI Costs By 10x — Over-reliance on large frontier models for all AI tasks leads to unnecessary costs and latency.
• The Death of the Monolithic Model: Why Future AI Systems Will Be Swarms, Not Giants — The future of AI systems lies in collaborative "swarms" of specialized models, not monolithic giants, driven by production economics and reliability.
• Full Sail on Asynchronous Inference — Asynchronous inference offers massive cost savings for LLM workloads by optimizing for throughput over real-time latency.
• Smart AI routing explained — Smart AI routing, not model size, is key to managing cost and risk in AI deployments.
• Presentation: The AI Gateway: Scaling Centralized Inference Across Decentralized Teams — AI model gateways provide a critical control layer to manage diverse AI inference across decentralized teams while maintaining centralized governance.
• The Minimill of AI — Distributing AI tasks between local and cloud models based on complexity significantly boosts efficiency and reduces reliance on hyperscalers.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Tomasz Tunguz.