Architecting Cost-Aware LLM Workloads with Model Router in Microsoft Foundry

2026-04-28 · Source: Microsoft Foundry Blog articles · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Intermediate, long

Summary

Microsoft Foundry's Model Router, released April 28, 2026, offers a platform-level solution for managing diverse LLM workloads by dispatching requests across up to 18 underlying LLMs from a single endpoint. This trained routing model analyzes prompt complexity and task type to select the most appropriate model, addressing common architectural challenges like overpaying for simple prompts or underperforming on complex ones. Architects can govern routing mode (Balanced, Quality, Cost), define model subsets for compliance and cost control, and specify deployment type and region. The platform handles real-time routing, automatic failover, data-zone enforcement, and underlying-model versioning. It supports standard chat completions, streaming, and tool use, with specific parameter handling for reasoning models and detailed monitoring via Azure portal metrics and cost analysis.

Key takeaway

For AI Architects designing GenAI platforms, Model Router in Microsoft Foundry simplifies multi-model dispatch, consolidating governance and observability. You should define routing modes and model subsets to align with your specific cost, quality, and compliance requirements, ensuring at least two models for failover. This approach centralizes LLM management, reducing application-layer complexity and providing clear cost attribution.

Key insights

Microsoft Foundry's Model Router intelligently dispatches LLM requests across diverse models for cost, quality, and governance.

Principles

• Routing decisions should be platform-level, not application-level.
• Model subsets enable granular control over compliance and cost.
• Effective context window is limited by the smallest model in the subset.

Method

Deploy Model Router with a routing mode (Balanced, Quality, Cost) and an optional model subset. Call it as a standard chat-completions endpoint, capturing `response.model` for attribution. Monitor performance and cost in Azure portal.

In practice

• Log `response.model` for per-request cost and routing analysis.
• Curate model subsets to manage compliance and context window.
• Use Cost mode for latency-sensitive, high-volume classification tasks.

Topics

• Model Router
• Microsoft Foundry
• LLM Workload Management
• Multi-Model Routing
• GenAI Architecture

Best for: AI Architect, MLOps Engineer, AI Engineer

Related on AIssential

• Token Spend Out of Control? The Case for Smarter Routing — Dynamically routing LLM agent requests to the cheapest capable model is crucial for controlling escalating token spend.
• Microsoft Foundry: An End-to-End Platform for Building, Governing, and Scaling AI — Microsoft Foundry unifies AI development and operations for secure, scalable enterprise AI adoption.
• Cross-Region Model Connectivity Options in Microsoft Foundry: Supported Patterns and Tradeoffs — Azure API Management enables governed, VNet-secured cross-region model access in Microsoft Foundry, extending agent and inference capabilities.
• Brick: Spatial Capability Routing for the Mixture-of-Models (MoM) Paradigm — Brick optimizes Mixture-of-Models deployment by spatially routing queries based on model capabilities and query difficulty to balance cost and quality.
• OpenRouter Now Processes More Than a Quadrillion Tokens a Year — OpenRouter provides a unified API for 400+ AI models, optimizing access, routing, and cost-quality balance for diverse tasks.
• Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption — Multi-model LLM serving requires schedulers to account for non-linear offloading sensitivity and high preemption costs across diverse models and hardware.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Microsoft Foundry Blog articles.