How Microsoft Ships AI Agents at Enterprise Scale

2025-12-15 · Source: ByteByteGo Newsletter · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Advanced, extended

Summary

Microsoft scales AI agents for enterprise use through its Foundry platform, supporting over 80,000 enterprises and powering internal tools like Microsoft 365 Copilot, which serves over 20 million users. The company emphasizes that a robust "harness" surrounding the AI model is critical for production, as prototypes often fail due to issues with data retrieval, tool calls, user handling, and quality drift. This harness comprises five layers: an Inference Layer supporting 11,000+ models, an Agent Runtime for orchestration, an Observability and Governance Layer (Foundry Control Plane), an Identity Layer extending enterprise systems like Entra, and a Context Layer. The Context Layer utilizes iterative "retrieval-as-a-subagent" via Microsoft IQ services (Foundry IQ, Fabric IQ, Web IQ, Work IQ) to provide accurate information and enable agents to take actions with proper identity and audit trails. Continuous, rubric-based evaluation, including an auto-improvement loop, ensures agent reliability and adaptation to changing conditions.

Key takeaway

For AI Engineers and MLOps teams deploying agents to enterprise production, prioritize building a comprehensive "harness" around your models. Your agent's success hinges on robust context retrieval, proper identity management, and continuous, rubric-based evaluation, not just model performance. Implement iterative retrieval-as-a-subagent patterns and extend existing identity systems to treat agents as first-class principals with audit trails. This approach ensures reliability, governance, and adaptability, preventing common production failures.

Key insights

The "harness" surrounding an AI model is paramount for enterprise-scale production agents, enabling reliable action and continuous improvement.

Principles

• Production agents require a robust "harness" around the model.
• Context retrieval must be iterative and agentic.
• Agents need distinct identities and audit trails for actions.

Method

Implement a five-layer agent harness: Inference, Runtime, Observability, Identity, and Context. Utilize iterative retrieval-as-a-subagent for context and continuous, rubric-based evaluation with auto-improvement.

In practice

• Extend enterprise identity systems for agent principals.
• Place guardrails at the tool boundary, not just model I/O.
• Draft evaluation rubrics from agent configuration and traces.

Topics

• AI Agents
• Enterprise AI
• MLOps
• Retrieval-Augmented Generation
• AI Governance
• Continuous Evaluation
• Microsoft Foundry

Best for: CTO, Machine Learning Engineer, AI Product Manager, AI Engineer, MLOps Engineer, AI Architect

Related on AIssential

• The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks — A structured five-pillar framework is essential for successfully deploying and managing enterprise-scale AI agents in production.
• Toward a Four-Layer Architecture for Self-Hosted Enterprise AI Harnesses — Architectural boundaries, not components, are key to safely deploying self-hosted enterprise AI agent harnesses.
• Runbooks + RAG: How I Gave My AI SRE Agent the Context It Was Missing — The gap between AI agent demos and production is context, best bridged by retrieval-augmented organizational knowledge.
• We Added Too Many Guardrails and Broke Our Own Agent, Our AI VP of Finance Found a Setting We’d Missed for 8 Years, and an Agent Is Now the One Renewing Your Software: The Agents #007 — Real-world AI agent deployments yield significant revenue gains but require careful management.
• Trace raises $3M to solve the AI agent adoption problem in enterprise — Providing AI agents with deep operational context is crucial for enterprise adoption and scalable deployment.
• QCon AI Boston: Production AI Moves Beyond Prompts to Platforms, Harnesses, and Evals — Production AI demands robust platform engineering, moving beyond prompt-centric development to focus on infrastructure, trust, and comprehensive evaluation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by ByteByteGo Newsletter.