Shipping Voice AI Agents to Production
Sub-800ms latency, barge-in, multilingual and interruption handling — what actually matters when voice AI meets real customers.
The latency budget is 800ms end-to-end
Humans expect a response inside ~1s. Above 1.2s the conversation feels broken. That leaves ~800ms for VAD → ASR → LLM → TTS → audio out. Every component has to be optimised — not just the model. Streaming ASR, first-token streaming from the LLM, and streaming TTS chunks are non-negotiable.
Barge-in and interruption
Users interrupt. If your agent can't stop mid-sentence, users hang up. Full-duplex audio, robust endpointing and turn-taking prediction separate demoware from production voice AI. We use lightweight models per-frame to detect user speech and cancel TTS instantly.
Multilingual and code-mixed reality
Indian, MENA and Southeast Asian customers speak code-mixed languages — Hindi-English, Arabic-English, Tagalog-English. Off-the-shelf ASR breaks. We use language-aware ASR with per-utterance language ID, and prompt the LLM to respond in the caller's language.
Grounding, safety and escalation
Voice agents can't afford hallucinations — there's no citation to point at. RAG grounding is mandatory. Confidence thresholds trigger warm-transfer to a human. Sensitive intents (billing disputes, safety, legal) route to humans by policy, not judgement.
Measuring what matters
Track: task completion rate, average turns to resolution, latency p95, barge-in success, escalation rate. CSAT alone is misleading — a fast, wrong resolution scores well until the follow-up complaint lands.
Key takeaways
- Target sub-800ms end-to-end latency with streaming at every stage
- Barge-in and interruption handling are table stakes
- Language ID per utterance is required for code-mixed markets
- RAG-ground every answer; escalate low-confidence intents by policy
- Track task completion, turns, latency and escalation — not just CSAT
T7 Research
Enterprise AI Research Group
T7 Research is the research arm of T7 Solution, focused on benchmarking LLMs, evaluating RAG patterns, and compiling implementation playbooks for enterprise technology leaders.