Voice AI Latency: A Complete Teardown
Every millisecond in the voice loop, from mic capture to first spoken token — and where sub-800ms voice agents actually save time.
Why 800ms is the magic number
Human turn-taking in natural conversation happens at 200–500ms of silence. Push a voice agent past 800ms of end-to-end latency and it stops feeling like a conversation and starts feeling like an IVR with a nicer voice. Every architecture decision downstream serves this budget.
The full latency budget, broken down
Microphone capture and voice-activity detection: 20–50ms. Speech-to-text streaming first token: 100–200ms. LLM first token: 200–500ms depending on model. Text-to-speech first audio: 100–200ms. Network round-trips: 50–150ms depending on colocation. Miss on any one of these and you blow the budget. Ship all of them well and you have a voice agent people don't hang up on.
Streaming everywhere or nothing works
Every stage — STT, LLM, TTS — must stream. Waiting for a complete transcript before calling the LLM adds hundreds of milliseconds. Waiting for a complete LLM response before starting TTS adds hundreds more. The pipeline is a river, not a bucket brigade.
Model choice is a latency decision
Frontier models with reasoning steps are wrong for voice. We standardise on models optimised for first-token latency — GPT-4o Realtime, Gemini Live, Claude Haiku on Bedrock — and offload complex reasoning to background jobs the agent references, not synchronously produces.
Colocation matters more than model
A 4o-mini call from the wrong region can be slower than a 4o call from the right one. Deploy inference next to the telephony provider. For India, that means Mumbai. For UAE, Dubai or Bahrain. For the US, Virginia or Oregon depending on carrier.
Interruption handling is a product feature
A voice agent that can't be interrupted mid-sentence is worse than a well-designed IVR. Barge-in — detecting user speech and cutting the agent's audio within 150ms — is a non-negotiable feature, not an edge case. It's also the difference between an agent that gets used and one that gets bypassed.
Where sub-800ms voice actually saves time
Authentication, appointment scheduling, order-status checks, simple troubleshooting, outbound qualification and reminders. Not: complex claims disputes, technical incident triage, negotiations. Voice AI is a scalpel, not a hammer — deploy it where it wins.
Key takeaways
- Design every stage of the voice pipeline against an 800ms budget
- Stream STT, LLM and TTS end-to-end
- Colocate inference with your telephony provider by region
- Ship barge-in as a first-class feature, not an edge case
T7 Research
Enterprise AI Research Group
T7 Research is the research arm of T7 Solution, focused on benchmarking LLMs, evaluating RAG patterns, and compiling implementation playbooks for enterprise technology leaders.