Voice Agents That Handle Interrupts - Chintan Agrawal and Daniel Wirjo, AWS

Voice Agent Technical Constraints - Human-to-human conversation turn-taking typically occurs within a 200-millisecond threshold, while latency exceeding 800 milliseconds degrades user experience, and 1.5-second delays often lead to user abandonment [7][8] - Current industry benchmarks for cascaded voice-to-voice pipelines show a best-measured response time of 755 milliseconds, which remains approximately 4 times slower than natural human interaction [8][9][46] - The total latency budget for standard cloud-based API setups ranges from 1,100 to 1,300 milliseconds, with Speech-to-Text (STT) and Large Language Model (LLM) processing consuming roughly two-thirds of this budget [37][39] Turn-Taking Architecture & Optimization - Voice Activity Detection (VAD) serves as the primary front-end component for turn-taking, though basic silence detection (Level 1) fails to distinguish between thinking pauses, incomplete thoughts, and completed sentences [11][19][20] - Advanced turn-taking strategies include Level 2 (STT-integrated endpointing) and Level 3 (local Smart Turn models), with the latter achieving 58.9% recall and 68.4% precision to improve responsiveness [21][22][25][26] - Effective interruption handling requires a pipeline capable of flushing downstream processes (TTS and LLM generation) within 50 milliseconds to ensure the system can immediately yield to user input [13][29] LLM Performance & Production Considerations - For voice applications, LLM performance is measured by "Time to First Token," with a target of under 700 milliseconds; models like Nemotron-3 Ultra (529 ms P50) and GPT-4 (536 ms P50) meet this requirement, though P95 tail latency spikes can reach 1.7 seconds or higher [40][41][43][44] - Multi-turn interactions often suffer from context degradation, where models may become overly verbose or ignore system prompts after 15 to 20 turns, necessitating context pruning or session resets [44][45] - False interruptions significantly increase user escalation rates, highlighting the need for sophisticated classification to differentiate between genuine interruptions and background noise or fillers [31][46]