Monitoring & Observability in Microsoft Foundry Part 2: Configuration and Operations

2026-07-09 · Source: Microsoft Foundry Blog articles · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Intermediate, long

Summary

This article, Part 2 of a series on Microsoft Foundry, details the configuration and operational patterns for robust AI observability, building on the conceptual framework of evaluation, monitoring, and tracing. It emphasizes connecting Azure Monitor Application Insights as the foundational telemetry layer. Key operational steps include enabling continuous evaluation of sampled production traffic for metrics like coherence, relevance, groundedness, and retrieval quality. The piece explains how to interpret the Agent Monitoring Dashboard to correlate quality, latency, failures, and token consumption, and how to use OpenTelemetry-based tracing to debug quality regressions. It also covers wiring alerts into Azure Monitor, integrating quality gates into CI/CD pipelines, enforcing guardrails via the Compliance workspace, and extending visibility with Defender for Cloud and Microsoft Purview for security and data governance. The article concludes by stressing the importance of monitoring cost, quota, and token usage alongside quality.

Key takeaway

For MLOps Engineers or AI Engineers deploying or managing AI agents in Microsoft Foundry, you must establish a comprehensive observability framework beyond basic uptime metrics. Prioritize connecting Application Insights, enabling continuous evaluation, and integrating OpenTelemetry-based tracing. Implement CI/CD quality gates and configure Azure Monitor alerts for critical signals like quality drops or cost spikes. Ensure compliance and security by leveraging Defender for Cloud and Purview, treating cost as a first-class operational constraint.

Key insights

Microsoft Foundry's AI observability integrates continuous evaluation, monitoring, tracing, security, and cost management for production agents.

Principles

• Observability requires continuous evaluation on live traffic.
• Correlate quality, latency, cost, and safety signals.
• Use consistent evaluation logic across environments.

Method

Configure Azure Monitor Application Insights as the telemetry backbone. Enable continuous evaluation on sampled production traffic. Use OpenTelemetry for tracing. Integrate alerts with Azure Monitor and quality gates in CI/CD.

In practice

• Connect Application Insights via "Operate > Admin > Connected resources".
• Enable continuous evaluation in "Build > Agents > Monitor".
• Implement CI/CD quality gates with evaluation scripts.

Topics

• Microsoft Foundry
• AI Observability
• Application Insights
• Continuous Evaluation
• OpenTelemetry Tracing
• CI/CD Quality Gates
• Azure Monitor

Best for: MLOps Engineer, AI Engineer, Director of AI/ML

Related on AIssential

• Monitoring & Observability in Microsoft Foundry — Production AI requires continuous, integrated observability beyond traditional APM to manage dynamic behavior and ensure quality, safety, and cost.
• AI Observability Starter Kit for Microsoft Foundry agents — Production-grade AI observability requires instrumented traces, automated quality and safety evaluations, and proactive alerting.
• Mind the Gap (In your Agent Observability) — Amy Boyd & Nitya Narasimhan, Microsoft — Effective AI agent observability requires continuous evaluation, monitoring, and automated optimization to manage non-determinism and ensure reliability.
• If You Can’t Observe Your Agents, You Don’t Own Them: A Deep Dive into Multi-Agent Observability… — Agent observability, enabled by OpenTelemetry and SigNoz, is crucial for debugging, cost management, and reliability of production AI agents.
• InsightFinder raises $15M to help companies figure out where AI agents go wrong — AI observability must encompass the entire tech stack, correlating data, models, and infrastructure for comprehensive diagnosis.
• Observability and human intuition in an AI world — Observability platforms and AI-driven incident resolution tools enhance debugging and operational efficiency.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Microsoft Foundry Blog articles.