Choose the right speech and voice model

2026-07-16 · Source: Microsoft Foundry Blog articles · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, extended

Summary

Microsoft Foundry offers a comprehensive guide for selecting the appropriate speech and voice model from its diverse portfolio, including Azure OpenAI, Voice Live API, Azure Speech in Foundry Tools, and Microsoft AI (MAI) speech models. The core recommendation is to choose models based on the system's required output, such as live conversation, transcription, translation, diarization, or synthetic voice, rather than by model name. The guide outlines specific scenarios, recommending Azure OpenAI Realtime API for direct speech-to-speech control, Voice Live API for managed voice agents with features like interruption handling, and Azure Speech for durable transcripts, custom voices, or privacy-sensitive workloads. It also details models like mai-transcribe-1.5 for low-cost multilingual STT and gpt-4o-mini-tts for styled speech generation, providing crucial details on latency, context limits (e.g., 32,000 input tokens for gpt-realtime), and pricing (e.g., mai-transcribe-1.5 at \$0.36/hour).

Key takeaway

For AI Architects designing voice-enabled applications, your model selection must prioritize the system's guaranteed output and user experience. Avoid production failures by matching specific needs like real-time conversation, durable transcription, or branded voice to the right Azure OpenAI, Voice Live, or Azure Speech model. Always prototype the transport, validate regional availability and pricing, and rigorously test critical failure modes before deployment. This ensures optimal performance and cost-efficiency.

Key insights

Choosing speech models requires aligning system guarantees (e.g., real-time conversation, transcription) with specific model capabilities.

Principles

• Select speech models by required system guarantee, not model name.
• Production failures often stem from misaligned model capabilities.
• Validate region, quota, pricing, and lifecycle before commitment.

Method

Prototype the transport first, then validate model availability (region, quota, pricing), and finally test critical failure modes like interruption or diarization.

In practice

• Use Azure OpenAI Realtime API for direct, low-latency speech-to-speech.
• Employ Voice Live API for managed voice agents with interruption handling.
• Leverage Azure Speech batch for durable, diarized transcripts of large files.

Topics

• Speech-to-Text
• Text-to-Speech
• Voice Agents
• Azure OpenAI
• Microsoft Foundry
• Real-time Speech
• Diarization

Best for: AI Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• Azure Speech at Build 2026: Powering Voice Agents with Real-Time and Life-like Experiences — Azure Speech updates enable rapid development of real-time, customizable, and production-ready voice agents.
• Angelos Perivolaropoulos of ElevenLabs at RAAIS 2026 — Real-world voice AI demands balancing high speech-to-text accuracy with critical low-latency performance for effective interactive systems.
• From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents — Converting text-based tool-calling benchmarks to audio reveals significant model- and task-dependent performance gaps.
• How to best approach NLP Agent Call Centre Training — Generic NLP models fail in call centers; domain adaptation through specific training data and model tuning is crucial.
• Issue #120 - Turn-based voice AI agents — Voice agents fall into turn-based (sequential) or streaming (overlapping) architectures, each with distinct trade-offs.
• Why Emotion Matters More Than Sound — Building production-grade voice AI agents requires overcoming significant application-layer challenges beyond basic LLM integration.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Microsoft Foundry Blog articles.