Intelligent sampling in Microsoft Foundry: the science behind selecting better production traces

2026-06-15 · Source: Microsoft Foundry Blog articles · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, long

Summary

Microsoft Foundry's intelligent sampling feature, released on June 15, 2026, employs a MinHash farthest-first diversity sampler to select production traces for agent evaluation and fine-tuning. This technique prioritizes broad coverage of agent behavior over mirroring production frequencies. Validation on WildChat showed +29.1% higher lexical diversity and +44.8% larger vocabularies compared to uniform-random sampling. Across five additional datasets, vocabulary gains ranged from +5.7% to +86.3%. An LLM judge (GPT-4.1) preferred diversity-sampled data 78% for evaluation and 71% for training. Fine-tuning gpt-4.1 with diversity-sampled data resulted in 40% lower training loss and +8.6pp higher token accuracy, with comparable final generation quality. The method uses hashing, adding zero per-token cost.

Key takeaway

For MLOps Engineers building and evaluating agents, utilizing Microsoft Foundry's intelligent sampling feature is crucial for curating high-coverage datasets. This approach, which prioritizes behavioral breadth over production frequency, will accelerate fine-tuning convergence and surface critical edge cases for robust evaluation. Consider its use for long-tail evaluation suites and quality-critical training, but stick to uniform sampling for production-distribution benchmarking.

Key insights

Diversity sampling using MinHash farthest-first traversal significantly enhances dataset lexical variety and LLM judge preference for agent evaluation and fine-tuning.

Principles

• Prioritize coverage over frequency.
• Diversity sampling accelerates training.
• Hashing enables cost-efficient selection.

Method

Microsoft Foundry's intelligent sampling combines MinHash signatures to estimate Jaccard similarity between traces and farthest-first traversal to greedily select the most diverse subset, running server-side without LLM calls.

In practice

• Use for agent evaluation datasets.
• Apply to fine-tuning data selection.
• Generate rubrics from diverse traces.

Topics

• Microsoft Foundry
• Intelligent Sampling
• MinHash
• Farthest-First Traversal
• Agent Evaluation
• Fine-tuning Data
• Lexical Diversity

Code references

• openai/summarize-from-feedback
• Azure/azure-sdk-for-python

Best for: NLP Engineer, Machine Learning Engineer, MLOps Engineer, AI Engineer

Related on AIssential

• Weasel: Out-of-Domain Generalization for Web Agents via Importance-Diversity Data Selection — Weasel enhances web agent generalization and efficiency through importance-diversity trajectory selection and state pruning.
• Benchmarks in Microsoft Foundry (preview): Standardized model and agent quality checks — Microsoft Foundry Benchmarks standardize AI model and agent evaluation by integrating open-source benchmarks into the development workflow.
• Foundry IQ: Improve recall by up to 54% with knowledge bases — Foundry IQ knowledge bases significantly improve agentic retrieval's evidence recall and answer quality while reducing token costs.
• When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines — Selector quality, not just team diversity, dictates multi-agent LLM pipeline performance.
• Building an Evaluation Harness for Production AI Agents: A 12-Metric Framework From 100+ Deployments — Robust evaluation infrastructure is critical for successful, compliant AI agent deployments in production.
• AiChemy: Next-Generation Agent with MCP, Skills and Custom Data for Drug Discovery — Databricks enables building custom multi-agent systems by integrating public and proprietary data sources.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Microsoft Foundry Blog articles.