Live demo of AssemblyAI's Universal-3.5 Pro Realtime Speech-to-Text model

2026-06-23 · Source: AssemblyAI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Intermediate, medium

Summary

AssemblyAI has released Universal-3.5 Pro, its latest real-time speech-to-text model, offering state-of-the-art accuracy across 19 languages with native code-switching. This model features a fully promptable interface, allowing users to provide context for improved vocabulary recognition, as demonstrated with medical terms like "ejection fraction of 35%" and "metoprolol succinate 50 mg" in a cardiology call, and proper capitalization of product names like "Bubble Gun 3000" and order IDs in a customer service scenario. A new "conversation context" feature dynamically updates the model configuration based on previous agent responses, crucial for disambiguating phonetically similar words like "C" and "Sí". Additionally, "voice focus" isolates the primary speaker, suppressing background noise in both near-field (headsets) and far-field (conference rooms) environments, with an adjustable suppression threshold.

Key takeaway

For voice agent developers building real-time transcription systems, Universal-3.5 Pro offers critical features to enhance accuracy and user experience. You should integrate contextual prompts for domain-specific vocabulary. Also, leverage conversation context to resolve phonetic ambiguities in agent-customer interactions. Applying voice focus with appropriate near-field or far-field settings will significantly improve transcription quality in noisy environments. This ensures your downstream processes receive clean, accurate text from the start.

Key insights

Contextual prompting and noise suppression significantly enhance real-time speech-to-text accuracy across diverse languages and challenging audio environments.

Principles

• Contextual prompting improves transcription accuracy.
• Dynamic agent context resolves phonetic ambiguities.
• Voice focus isolates primary speaker from noise.

Method

The model uses a promptable interface to provide domain, topic, or scenario context. Conversation context dynamically updates configuration with previous agent turns. Voice focus applies near/far field noise suppression with a threshold.

In practice

• Provide domain prompts for specialized vocabulary.
• Integrate conversation context for voice agent interactions.
• Utilize voice focus for noisy audio environments.

Topics

• Real-time Speech-to-Text
• Universal-3.5 Pro
• Multilingual Transcription
• Code-Switching
• Contextual Prompting
• Voice Focus

Best for: AI Architect, Machine Learning Engineer, CTO, AI Engineer, NLP Engineer, MLOps Engineer

Related on AIssential

• Prompt Engineering Workshop: Universal-3 Pro — Universal 3 Pro offers customizable speech-to-text via natural language prompts, enhancing transcription accuracy and control.
• Voice agent APIs in 2026, compared: which one actually hears your users? — Production voice agents require robust accuracy on critical data, not just clean speech, to succeed in real-world conditions.
• Code Switching in Real-Time | Universal-Streaming Speech-to-Text — Assembly AI's universal model enables real-time, zero-latency code-switched speech-to-text across six languages.
• TAI #198: Real-Time Speech AI Gets Serious: Google and OpenAI Race to Own the Voice Layer — Real-time speech AI is maturing into deployable infrastructure with significant advancements in quality, speed, and cost efficiency.
• Angelos Perivolaropoulos of ElevenLabs at RAAIS 2026 — Real-world voice AI demands balancing high speech-to-text accuracy with critical low-latency performance for effective interactive systems.
• Why Emotion Matters More Than Sound — Building production-grade voice AI agents requires overcoming significant application-layer challenges beyond basic LLM integration.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AssemblyAI.