Quoting Luke Curley

2026-05-09 · Source: Simon Willison's Weblog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, quick

Summary

Luke Curley, in response to OpenAI's article on low-latency voice AI, argues that WebRTC's design is fundamentally problematic for large language model (LLM) prompt delivery. He highlights that WebRTC is engineered to aggressively degrade and drop audio packets under poor network conditions to maintain low latency, a feature beneficial for real-time conference calls. However, for LLM interactions, users prioritize prompt accuracy over immediate delivery, even if it means waiting an additional 200ms. Curley notes that retransmitting WebRTC audio packets within a browser is practically impossible, citing past difficulties at Discord, because the implementation is hard-coded for real-time latency, which can lead to "garbage prompts" and subsequent "garbage responses" from LLMs.

Key takeaway

For NLP Engineers designing voice AI systems, you should critically evaluate WebRTC's suitability for LLM prompt transmission. Its inherent design to drop packets for low latency can compromise prompt accuracy, leading to suboptimal LLM responses. Consider alternative protocols or custom solutions that prioritize data integrity and retransmission capabilities over strict real-time delivery for user inputs, especially when prompt quality directly impacts output utility.

Key insights

WebRTC's real-time optimization degrades LLM prompt accuracy, prioritizing speed over data integrity.

Principles

• Latency vs. Accuracy is a critical trade-off.
• Real-time protocols may harm non-real-time data.
• Packet retransmission is vital for data integrity.

In practice

• Avoid WebRTC for critical LLM prompt delivery.
• Prioritize data integrity over minimal latency.
• Evaluate protocol suitability for specific use cases.

Topics

• WebRTC
• OpenAI Voice AI
• Low Latency
• Prompt Accuracy
• Audio Packet Loss

Best for: Machine Learning Engineer, NLP Engineer, CTO, AI Engineer, MLOps Engineer, AI Architect

Related on AIssential

• Voice Live API now supports WebRTC (Preview) — WebRTC integration in Voice Live API enables low-latency, real-time voice interactions for AI agents.
• How I Improved Speech-to-Text Accuracy — A two-pass LLM post-processing method effectively corrects common speech-to-text transcription errors.
• The Secret to Eliminating Latency in Voice AI 🔧 — High-volume voice agents benefit from scripting and redundancy to meet precise business communication needs.
• I Benchmarked 30+ Voice AI Engines and Built a Real-Time Translator Faster Than Google Meet — Optimizing real-time voice AI requires careful component selection and protocol choice to minimize latency and cost.
• From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents — Converting text-based tool-calling benchmarks to audio reveals significant model- and task-dependent performance gaps.
• Building Real-Time Speech Translation with AI Avatars with Azure Speech Services — Real-time AI avatar speech translation enhances global communication by delivering personalized, immersive multilingual experiences.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Simon Willison's Weblog.