The Voice Agent Latency Budget: Where Your Milliseconds Actually Go

Voice agents live or die on latency. Everyone nods at this in the abstract, and then almost everyone budgets for it wrong — because the latency that kills a voice agent isn't in the place you'd guess.

I spent three years shipping voice agents and live AI avatars in production — LiveKit for transport, Pipecat for the voice pipeline, Deepgram for ASR, ElevenLabs for TTS, Anthropic models for the reasoning — and the most common reason a deployment felt "off" was almost never the quality of the answers. It was the pause before them. Users will forgive a slightly wrong response delivered in a natural conversational beat. They will not forgive two seconds of dead air, no matter how good the words are when they finally arrive.

This post is the latency budget I wish someone had handed me on day one.

The budget

Human turn-taking is fast. In natural conversation, the gap between one speaker finishing and the other starting is typically a few hundred milliseconds. That's the bar your agent is measured against, whether you like it or not.

As industry framing: aim for roughly 500–800ms end-to-end — from the moment the user stops speaking to the moment your agent's audio starts playing. Under ~800ms feels responsive. Past a second, it feels like talking to a system. Past two seconds, users start saying "hello?" and repeating themselves — and now you have overlapping speech corrupting your transcript, and everything downstream degrades.

Notice what that budget is not: it's not the time to generate a full response. It's time-to-first-audio. That distinction drives every design decision that follows.

Where the milliseconds hide

The naive mental model is a pipeline with three costs: ASR, LLM, TTS. The real pipeline has five or six, and the biggest one is invisible in that model.

1. Endpointing — the silent budget killer. Before your agent can respond, something has to decide the user has finished speaking. The default approach is a voice-activity-detection timer: wait for N milliseconds of silence, then declare the turn over. Set that timer to 700ms and you've burned your entire budget before a single model has run. Set it to 200ms and you'll cut people off mid-sentence every time they pause to think — which feels far worse than being slow.

This is the trade-off nobody tells you about: endpointing is a bet on whether the user is done, and every millisecond you shave off it raises the odds of interrupting them. In practice you tune it per use case (short confirmations can endpoint aggressively; open-ended questions can't), and increasingly you supplement raw silence detection with semantic signals — a transcript ending in "and, um" is probably not a finished turn, whatever the VAD says.

2. ASR finalization. Streaming ASR gives you interim transcripts continuously, but the final transcript — the one that's safe to act on — lands after endpointing resolves. The trick that buys real time: don't wait for final. When the interim transcript has been stable for a beat, start the LLM speculatively. If the final transcript matches, you've overlapped ASR tail latency with LLM latency for free. If it doesn't, you throw the speculative run away. You'll waste some tokens; that's a good trade.

3. LLM first token. For voice, total generation time is nearly irrelevant — time-to-first-token is everything, because you're streaming the output into TTS anyway. The levers: keep the prompt lean (a bloated system prompt taxes every single turn), use prompt caching on the stable prefix, and use the smallest model that passes your evals for the conversational layer. A frontier model with 900ms first-token latency is worse for voice than a smaller one at 300ms, for most turns.

4. TTS start. Same story: what matters is time-to-first-audio-byte, not time to synthesize the whole response. Stream text into the TTS as the LLM produces it, and chunk on natural boundaries — first clause or first sentence — so synthesis can begin after a handful of words. A well-tuned pipeline has the agent speaking its first phrase while the LLM is still writing the third sentence.

5. Transport. WebRTC exists for a reason. It's built for exactly this: low-latency, jitter-tolerant, bidirectional audio. If you're bridging to telephony, you inherit the phone network's latency floor on top of everything above. And region matters — a pipeline that's snappy with everything co-located in one region gets noticeably worse when your media server and your inference are an ocean apart. Put them next to each other.

Stream everything, overlap everything

The single principle underneath all of the above: a voice pipeline is not a sequence, it's a set of overlapping streams. ASR streams into the LLM before the turn is final. The LLM streams into TTS before the response is complete. TTS streams to the user before synthesis is done.

Every place you wait for a stage to finish before starting the next, you're adding its full duration to the user's silence. Every place you overlap, you're hiding one stage's latency inside another's. The difference between a demo pipeline and a production pipeline is mostly this — the demo runs the stages in order; production runs them as a cascade.

Barge-in: the feature that proves you're real-time

Humans interrupt each other constantly, and they will interrupt your agent too — especially when it's reading out something long. If your agent keeps talking over them, the illusion of conversation collapses instantly.

Handling barge-in properly means three things:

Stop fast. When the user starts speaking, kill playback within a perceptual instant — flush the audio queue, cancel in-flight TTS, cancel the LLM generation feeding it.
Track what was actually said. This is the subtle one. The LLM generated five sentences; the user heard two and a half. If your conversation history records the full response, the model now believes it said things the user never heard, and the dialogue drifts. Truncate the assistant turn in history to what was actually played.
Distinguish interruption from backchannel. "Mm-hmm" and "right" are the user telling you to keep going, not to stop. Cutting the response every time someone makes an agreement noise is its own failure mode.

Barge-in is where the pipeline's architecture gets stress-tested, because it requires every stage to be cancellable mid-flight. If you built the cascade properly, it falls out naturally. If you built a sequence, it's a rewrite.

What users actually perceive

After all the measurement, the thing that matters is perception, and perception is not a stopwatch.

Consistency beats raw speed. An agent that responds in 700ms every single time feels better than one that averages 500ms but occasionally takes two seconds. Humans lock onto conversational rhythm quickly; variance is what breaks the spell. This is why you optimize p95, not the mean — the slow turns are the ones users remember.

Acknowledgment buys time. When a turn genuinely needs a slow operation — a tool call, a lookup — a brief, natural acknowledgment ("let me check that") resets the user's clock. It converts dead air into a normal conversational move. Use it honestly and sparingly; an agent that says "one moment" before every turn is telegraphing its own slowness.

The first turn sets the anchor. Users calibrate on the opening exchange. Nail the greeting latency — it's the cheapest turn you have, there's no ASR or reasoning needed — and you've bought goodwill for the rest of the call.

I ended up going deep enough on this problem to write a paper on voice AI latency, and the conclusion after all the digging is almost embarrassingly simple: measure end-to-end from the user's perspective, and be suspicious of any stage-level metric that isn't tied back to time-to-first-audio.

The takeaway

The latency budget for a production voice agent is roughly 500–800ms of silence, and it gets spent in places the architecture diagram doesn't show: the endpointing timer, the ASR finalization tail, first-token wait, TTS spin-up. You don't win it back with a faster model. You win it back by overlapping every stage, streaming everything, making the whole cascade cancellable for barge-in, and optimizing the p95 a human actually hears.

Latency isn't a metric on your dashboard. In voice, it's the interface. Everything else is negotiable; the pause before your agent speaks is not.

Related reading

Building Production AI Agents in 2026
Designing the Agent-to-Human Handoff
See the voice AI systems I've shipped and the research behind them.