OpenAI Outlines WebRTC Architecture for Low-Latency Voice AI at Scale

2026-05-20 · Source: InfoQ · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Intermediate, quick

Summary

OpenAI recently detailed its adapted WebRTC architecture for achieving low-latency voice AI at global scale, replacing a conventional media termination model. This new design, better suited for Kubernetes and cloud load balancers, addresses constraints like global reach, fast connection setup, and stable media round-trip times. The architecture employs a relay-transceiver split: lightweight relays handle incoming packets and forward them, reducing public UDP exposure and keeping media routing close to users. A separate transceiver layer manages all stateful WebRTC machinery, including ICE negotiation, DTLS handshakes, and SRTP encryption. This separation concentrates complexity in the transceiver, preventing its duplication across backend services or custom client behavior. This approach is optimized for OpenAI's predominantly 1:1 user-to-model sessions, unlike SFU designs common in multi-party systems, and supports products like ChatGPT voice and the Realtime API.

Key takeaway

For AI Architects or MLOps Engineers building interactive media systems, you should consider adopting a relay-transceiver WebRTC architecture. This approach, demonstrated by OpenAI, allows you to centralize complex session state while distributing lightweight, stateless relays globally, significantly improving latency and scalability for 1:1 user-to-model voice AI. Prioritize concentrating protocol complexity in a dedicated layer rather than spreading it across backend services or client logic.

Key insights

OpenAI's WebRTC architecture uses a relay-transceiver split to achieve low-latency voice AI at global scale for 1:1 user-model sessions.

Principles

• Preserve protocol behavior at the edge.
• Keep hard session state in one place.
• Move scaling complexity to a thin routing layer.

Method

The proposed method involves splitting WebRTC responsibilities into a lightweight, stateless relay for packet forwarding and a stateful transceiver for ICE, DTLS, SRTP, and session lifecycle management.

In practice

• Implement a stateless relay for media routing.
• Centralize WebRTC session state in a transceiver.
• Optimize for 1:1 user-model interactions.

Topics

• WebRTC
• Low-Latency AI
• Voice AI
• Kubernetes
• Cloud Architecture
• Real-time Systems

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Architect, AI Engineer, MLOps Engineer

Related on AIssential

• How OpenAI delivers low-latency voice AI at scale — OpenAI's WebRTC rearchitecture uses a split relay-transceiver model for scalable, low-latency voice AI.
• Building Scalable AI Voice Agents: Architectures, Latency & Best Practices — Voice agents are fundamentally a latency problem disguised as an AI problem, where speed and natural conversation outweigh raw model intelligence.
• 🔴 LIVE: Google’s $15B AI Hub Faces Protests | AI Chip Crisis & G7 AI Battle | Front Page — The global AI boom is creating significant infrastructure, supply chain, and governance challenges, while also driving innovation in specialized areas like voice AI.
• Taming Voice Complexity with Dynamic Ensembles at Modulate — Ensemble Listening Models (ELMs) use dynamically routed, specialized small models for cost-effective, accurate Voice AI.
• Building the AI Grid with NVIDIA: Orchestrating Intelligence Everywhere — AI-native services are exposing a new bottleneck in AI infrastructure, shifting the challenge from peak training throughput to delivering deterministic inference at scale with predictable latency and…
• Sakana AI Introduces KAME: A Tandem Speech-to-Speech Architecture That Injects LLM Knowledge in Real Time — KAME integrates LLM knowledge into real-time speech generation via asynchronous parallel processing, achieving high quality with zero latency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by InfoQ.