Oleksandr Kuzmenko, EditorPublished 8 min readAI-assisted · editor-reviewed
TL;DR
- WebRTC NetEQ aggressively drops audio packets delayed by >30ms, corrupting early prompt context for speech-to-text models.
- Audio consumes only 8KB per second at 64Kbps Opus, allowing seconds of retransmission time over QUIC before prompt context is required.
- TTS pipelines over WebRTC require artificial sleeping to avoid timing corruption, whereas MoQ buffers timestamped frames directly in RAM.
Key facts
- Opus Audio Bandwidth
- 8KB per second (64Kbps)
- WebRTC Drop Threshold
- >30ms packet latency
- Opus WebRTC Packet Rate
- ~50 packets per second
- QUIC Retransmission Packet Equivalent
- ~6 packets per second
- Client RAM Requirement for 2 Min Audio
- ~1MB
The Flaw of Human-Centric Jitter Buffers
WebRTC is bound to libwebrtc, the Google-maintained C++ monolith powering Google Meet. In meeting software, any audio arriving late causes unpleasant talk-over overlap. NetEQ, WebRTC's audio engine, actively adapts playback speed to network jitter but discards packets delayed past human tolerance thresholds (roughly 30ms). Audio retransmission exists in WebRTC only via custom Session Description Protocol (SDP) munging, yet the NetEQ jitter buffer refuses to wait for retransmitted packets.
When a voice AI client hits brief packet loss, WebRTC discards early user words. The speech-to-text (STT) layer receives truncated audio, forcing the model to infer context from corrupted sentences or ask the user to repeat the prompt.
Why LLM Latency Budgets Differ
Voice agents do not need real-time audio consumption during the user's turn. The model must ingest the entire prompt before issuing a meaningful response.
- Audio bandwidth is trivial: 1 second of Opus at 64Kbps requires 8KB, delivered across roughly 50 raw packets.
- QUIC recovery is fast: Those dropped frames condense into roughly 6 QUIC packets.
- Zero latency penalty: Since the user is still speaking the end of the sentence, the agent network stack has multiple seconds to retransmit lost packets prior to final model inference.
Solving Text-to-Speech Bursting with Media over QUIC
WebRTC renders media based on network arrival times rather than embedded presentation timestamps. Consequently, server-side text-to-speech (TTS) engines that generate audio faster than 1x speed must insert explicit sleep cycles between packets to avoid overwhelming browser playback.
Protocols like MoQ (Media over QUIC) leverage client-side memory buffers. Because two minutes of uncompressed audio occupies merely 1MB of client RAM, servers can stream generated voice responses at maximum network bandwidth, storing frames in local buffers and rendering them precisely according to playback timestamps.
When to use
- Real-time voice AI agents streaming bidirectional audio over unstable mobile, cellular, or satellite connections.
- Speech pipelines where dropping early audio syllables forces the LLM into context hallucinations or repeated turns.
When NOT to use
- Browser-only web clients where WebTransport and QUIC protocols are blocked by strict corporate firewalls.
- Pure human-to-human teleconferencing applications where human audio overlap penalty outweighs packet retransmission.
What to do today
- Evaluate Media over QUIC or WebSocket-based chunk streaming if your voice agent suffers from dropped words on mobile connections.
- Verify whether your voice pipeline uses NetEQ-based WebRTC or explicitly buffers audio frames by presentation timestamps before STT inference.
- Check audio client memory allocation to buffer full TTS playback frames (roughly 1MB per two minutes of Opus audio).
Step names are taken from What to do today.
Step names are taken from What to do today. Evaluate Media over QUIC or WebSocket-based chunk streaming if your voice agent suffers from dropped words on mobile connections.. Verify whether your voice pipeline uses NetEQ-based WebRTC or explicitly buffers audio frames by presentation timestamps before STT inference.. Check audio client memory allocation to buffer full TTS playback frames (roughly 1MB per two minutes of Opus audio).
- Evaluate Media over QUIC or WebSocket-based chunk streaming if your voice agent suffers from dropped words on mobile connections.
- Verify whether your voice pipeline uses NetEQ-based WebRTC or explicitly buffers audio frames by presentation timestamps before STT inference.
- Check audio client memory allocation to buffer full TTS playback frames (roughly 1MB per two minutes of Opus audio).
What the community says
Size is the tough part. These models require 256GB+ of vram, but other then that I think it's possible!
This is for communicating with agents and inference, not just dictation.
Until then, anyone (including OpenAI) trying to use WebRTC for human<->bot communication may run into issues.
