What Causes Latency in AI Voice Agents
Latency in an AI voice agent is the delay between the moment a caller finishes speaking and the moment they hear the agent respond. It is not produced by any single component. It is the sum of several sequential steps: detecting that the caller has stopped speaking, converting their speech to text, running a language model to decide what to say, converting the reply back into speech, and moving audio across the phone network in both directions. Each step adds time, and the total is what a caller experiences as responsiveness or as an awkward pause.
Two facts shape everything about this problem. First, human conversation has very little tolerance for delay: people leave gaps of roughly 200 milliseconds between turns, and once a response takes noticeably longer than that, a call starts to feel unnatural. Second, the largest delays usually come from model inference and from deciding when the caller has finished, not from the speech recognition and synthesis that most people assume are the bottleneck. This article breaks down every contributor, separates the AI processing layer from the telecom transport layer, and explains what actually reduces the total.
Quick facts
| Stage | What it does | Typical contribution |
|---|---|---|
| Turn detection | Decides the caller has stopped speaking | 150 to 300 ms (much higher with naive silence timers) |
| Speech recognition (STT) | Converts caller audio into text | 100 to 300 ms |
| Model inference (LLM) | Generates the reply, measured to first token | 250 to 1000 ms |
| Speech synthesis (TTS) | Converts the reply text into audio | 100 to 200 ms to first audio |
| Telecom transport | Carries audio across the network | 30 to 80 ms each direction |
| Target voice-to-voice total | End of caller speech to start of reply | Under about 800 ms for natural conversation |
Why latency matters for voice
Voice is unforgiving in a way that text chat is not. In a chat interface, a one-second delay is invisible. In a phone conversation, it is a held breath.
Research on human turn-taking finds that speakers hand off with gaps of around 200 milliseconds, and human-computer interaction studies find that delays below roughly one second feel fluid while longer pauses begin to feel disruptive. In practice, voice agent builders treat somewhere under about 800 milliseconds of voice-to-voice latency as the threshold for a conversation that feels natural. Below that, callers relax. Above it, they start talking over the agent, repeating themselves, or assuming the call has dropped, and above roughly 1.5 seconds the conversation feels broken. Latency is therefore not a performance nicety; it directly drives whether callers stay on the line.
The AI voice pipeline, stage by stage
A voice agent is a chain of processing steps, and the total latency is the sum of the time each step consumes. Knowing where the milliseconds actually go is the first step to reducing them.
Caption: One conversational turn passes through turn detection, speech recognition, model inference, and speech synthesis, wrapped by telecom transport in both directions. Model inference and turn detection are usually the largest contributors.
Turn detection and endpointing
Before the agent can respond, it has to decide that the caller has actually finished, not just paused mid-sentence. This is turn detection, also called endpointing or end-of-utterance detection, and it is one of the two biggest and most overlooked sources of latency.
The naive approach waits for a fixed period of silence, often one to 1.5 seconds, before treating the turn as complete. That single choice can consume more than the entire target budget before any AI work begins. Better systems use models that detect the end of a turn far faster, often in the 150 to 300 millisecond range, and the most capable ones use semantic endpointing that considers whether the sentence sounds complete rather than relying on a rigid timer. Tuning this is a balance: too aggressive and the agent interrupts the caller; too conservative and every reply lags.
Speech recognition
Speech-to-text, or STT, converts the caller’s audio into text the model can read. Modern streaming recognition is fast, typically adding 100 to 300 milliseconds, because it processes audio continuously as the caller speaks rather than waiting for the utterance to end. Contrary to common assumption, STT is usually not the bottleneck in a well-built stack.
Model inference
The language model reads the recognized text plus the agent’s instructions and generates a response. What matters for perceived latency is not how long the full answer takes to generate, but the time to first token: how quickly the model produces the beginning of its reply, since synthesis can start from there. This is the single largest variable in most stacks, ranging from around 250 milliseconds for a small, fast model to well over a second for a large one. Model choice is therefore one of the most consequential latency decisions, and a smaller model with low time-to-first-token often serves a phone call better than a larger, slower one, with complex reasoning offloaded only when genuinely needed.
Speech synthesis
Text-to-speech, or TTS, converts the reply into audio. As with STT, the figure that matters is time to first audio, not the time to render the whole sentence, because the agent can begin speaking as soon as the first chunk is ready. Streaming TTS commonly produces first audio in 100 to 200 milliseconds, and like recognition, it is rarely the main bottleneck.
Orchestration
Orchestration is the layer that stitches turn detection, STT, the model, and TTS together, manages barge-in when a caller interrupts, and handles function calls. It does not appear as a separate box in most latency budgets, but it introduces delay in a subtle way: every boundary between separately hosted services adds a network hop, and a stack assembled from a speech vendor, a model vendor, a synthesis vendor, and a carrier accumulates delay at each handoff. This is why architecture, and specifically how tightly these components are coupled and colocated, often matters as much as the raw speed of any one model.
The telecom transport layer is separate from the AI
A crucial distinction, and a common source of confusion, is that telecom transport and AI orchestration are different layers with different owners. The AI layer covers turn detection, recognition, inference, and synthesis. The transport layer covers everything that moves the audio: the SIP signaling that sets up the call, the RTP stream that carries the media, the codec, jitter buffering, carrier routing, and the connection to the public phone network. Conflating the two leads to misdiagnosis, because tuning a model will never fix a transport problem, and optimizing routing will never fix a slow model.
Transport contributes its own latency. Audio travels as a stream of small packets, usually 20 milliseconds of audio each, and a network round trip typically adds 30 to 80 milliseconds depending on geography and carrier before any AI work happens at all. Jitter buffers, which smooth out uneven packet arrival, add delay equal to their depth. Codecs add a small amount, from a negligible amount for G.711 to more for heavily compressed codecs. Geography compounds all of it: a call that crosses continents, or that hops between poorly connected carriers, accumulates delay on every leg.
The telecom industry has defended this budget for decades. The ITU-T G.114 recommendation specifies no more than about 150 milliseconds of one-way transmission delay for good interactive voice quality. An AI voice agent has to fit its entire recognition, inference, and synthesis pipeline inside and around a transport budget that was already tight before AI was added, which is why keeping the transport leg short is not optional.
Streaming: overlapping the stages
The most effective way to cut latency is to stop running the stages one after another and start overlapping them. In a sequential pipeline, each step waits for the previous one to finish, so the delays add end to end. In a streaming pipeline, recognition runs while the caller is still speaking, the model begins as soon as the text is ready, and synthesis begins on the model’s first tokens, so the reply starts long before the full response has been generated.
Caption: Running stages sequentially adds every delay end to end, pushing first audio out to around 1200 ms. Overlapping the stages through streaming lets the reply start around 650 ms after the caller stops.
The savings are large. Streaming recognition can save 100 to 200 milliseconds by not waiting for the utterance to end, streaming synthesis can save 200 to 400 milliseconds by playing audio before the full sentence is rendered, and streaming the model output lets synthesis start while generation continues. Combined, moving from a sequential to a streaming architecture commonly removes 300 to 600 milliseconds from the total, often the difference between a call that feels natural and one that does not.
How to reduce AI voice agent latency
Several levers reduce total latency, in rough order of impact:
- Stream every stage. Recognition, inference, and synthesis should all emit partial results rather than waiting for the previous stage to complete. This is usually the single biggest win.
- Fix turn detection. Replace fixed silence timers with fast end-of-utterance detection, ideally semantic, so the agent reacts promptly without cutting the caller off.
- Choose the model for the job. Favor a model with low time-to-first-token for conversational turns, and reserve larger models for the specific moments that need them.
- Colocate and shorten hops. Keep recognition, inference, synthesis, and orchestration close together, and deploy them near the caller, so audio does not cross regions between every stage.
- Keep the transport leg short. Use regional carrier routing and direct interconnection so the audio path to and from the phone network stays brief. This is a telecom decision, distinct from the AI stack.
- Use graceful interruption. A tightly coupled pipeline that can cleanly abort a response lets you tune endpointing more aggressively, recovering time, because a premature start can be cancelled without a jarring stutter.
Common mistakes
- Blaming STT or TTS. In a modern stack these are fast. The time almost always hides in turn detection and model time-to-first-token. Optimizing the wrong stage wastes effort.
- Using fixed silence timeouts. Waiting a full second of silence to confirm the turn spends most of the budget before any processing starts.
- Measuring full generation instead of first token or first audio. What the caller feels is when the reply starts, not when it finishes. Budget around first token and first audio.
- Ignoring vendor hops. A pipeline stitched across four separately hosted services pays a network penalty at each boundary. Colocation and tight coupling matter as much as model speed.
- Confusing transport delay with AI delay. A call that lags because it routes across the world will not be fixed by changing models, and a slow model will not be fixed by better routing. Diagnose the two layers separately.
- Optimizing latency into unnaturalness. Pushing endpointing too aggressively makes the agent interrupt. The goal is natural timing, not the lowest possible number.
Where didlogic fits
didlogic operates the telecom transport layer of an AI voice agent, not the AI layer. The recognition, model, and synthesis, along with turn detection and orchestration, run inside your AI voice platform, such as Vapi, Retell, LiveKit, or ElevenLabs. didlogic provides the DID numbers, SIP trunking, and carrier routing that carry the caller’s audio to that platform and the agent’s audio back to the caller.
That means didlogic influences the transport portion of the latency budget: the network path between the caller and your AI platform. Regional points of presence and direct carrier interconnection keep the audio path short before it reaches your platform’s media edge, which matters because transport delay is added on top of, and cannot be recovered by, the AI pipeline. What didlogic does not do is speech recognition, language modeling, or speech synthesis; those, and the turn-detection and orchestration decisions that dominate the AI side of the budget, remain with your platform. The practical implication is that latency work has two independent fronts: tune the AI pipeline inside your platform, and keep the carrier path short and direct at the infrastructure layer. You can read more about the infrastructure side on the didlogic AI voice page and the SIP trunking for developers page.
Frequently asked questions
What is a good latency target for an AI voice agent?
Aim for voice-to-voice latency under roughly 800 milliseconds, measured from the moment the caller stops speaking to the moment the agent’s audio begins. Below that, conversation feels natural. Above about 1.5 seconds, callers start to feel the call is broken.
Which part of the pipeline usually causes the most delay?
Turn detection and model inference, specifically the time to first token. Speech recognition and synthesis are typically fast in modern stacks. If you are chasing latency, start with how quickly the agent detects the end of a turn and how fast the model produces its first token.
Does streaming really make a difference?
Yes, and it is usually the biggest single improvement. Overlapping recognition, inference, and synthesis instead of running them one after another commonly removes 300 to 600 milliseconds from the total by letting the reply start before the full response is generated.
Is call latency the fault of the AI platform or the carrier?
Both, in different ways, which is why they should be measured separately. The AI platform owns recognition, the model, synthesis, turn detection, and orchestration. The carrier and transport layer own the network path, SIP, RTP, codec, and routing. A slow model and a long network route are different problems with different fixes.
Why does a call to a distant region feel slower even with a fast model?
Because telecom transport adds latency independent of the AI. Every network hop and long-distance route adds delay before and after the AI does any work, and the ITU-T G.114 guideline of about 150 milliseconds of one-way transmission delay is easy to exceed on a poorly routed international call. Keeping the carrier path short and regional addresses this.
Does the codec affect latency?
A little. Codecs add a small, mostly fixed amount of processing delay, from negligible for G.711 to more for heavily compressed codecs, and jitter buffers add delay to smooth packet arrival. These are modest compared with model inference and turn detection, but they are part of the transport budget.
Continue learning
Free trial is available for IT infrastructure managers and developers.