Voice agents (realtime)

A text agent can take three seconds to think and nobody minds. A voice agent cannot: if there’s more than roughly half a second of silence after you stop speaking, the conversation feels broken — stilted, robotic, wrong. Real-time voice is therefore its own engineering problem, where the agent must listen, transcribe, think, and speak continuously, all streaming, all under a brutal latency budget.

The cascaded pipeline

The classic voice agent is a cascade of components, each streaming into the next:

• VAD (voice activity detection) decides when the user is speaking, and turn detection / endpointing decides when they’ve finished (the hardest social cue to get right — too eager and you cut them off, too patient and it feels laggy).
• STT (speech-to-text) transcribes, streaming partial results.
• The LLM generates the reply.
• TTS (text-to-speech) synthesizes audio, streaming it out.
• Barge-in: if the user starts talking while the agent is speaking, it must stop immediately and listen — interruption handling is non-negotiable for a natural feel.

The latency budget forces streaming

The whole game is end-to-end latency — target roughly 500–800 ms from when you stop talking to when you hear a reply. The components add up fast, so you cannot run them sequentially. The trick is to stream and pipeline: feed partial transcripts into the LLM before the user even finishes, and stream the LLM’s tokens straight into TTS, so the user hears audio at the model’s first token, not after the whole response is written:

vad, stt, llm_ttft, llm_full, tts_first = 100, 150, 250, 900, 120   # ms

# Naive: wait for the FULL LLM response, then synthesize all the speech.
naive = vad + stt + llm_full + tts_first
# Streaming: stream LLM tokens into TTS; first audio plays at the LLM's first token.
streamed = vad + stt + llm_ttft + tts_first

print(f"naive (wait for full response): {naive} ms")
print(f"streamed (first audio at TTFT): {streamed} ms")
print(f"target <800 ms -> naive {'FAILS' if naive>=800 else 'ok'}, streamed {'ok' if streamed<800 else 'FAILS'}")

naive (wait for full response): 1270 ms
streamed (first audio at TTFT): 620 ms
target <800 ms -> naive FAILS, streamed ok

Waiting for the full response blows the budget at 1270 ms — unusable. Streaming, the user hears the first words after 620 ms because TTS starts on the LLM’s first token. That single change is the difference between a demo and a product.

Cascaded vs speech-to-speech

There are two architectures, and the field is shifting between them:

• Cascaded (STT → LLM → TTS). Modular — swap any component, pick the best STT and TTS — but latency stacks across stages and the text bottleneck throws away paralinguistics: tone, emotion, and emphasis are lost when speech becomes plain text.
• Speech-to-speech (realtime multimodal models — GPT Realtime, Gemini Live). Audio goes in and audio comes out of a single model. Lower latency (no cascade), preserves prosody and emotion, and handles interruptions natively. It’s the frontier direction, at the cost of less modularity and control.

In one breath

• Voice agents need a reply in roughly 500–800 ms or the conversation feels broken — real-time is the defining constraint.
• The cascaded pipeline streams VAD → STT → LLM → TTS, with turn detection (endpointing) and barge-in (stop when interrupted) as the hard social mechanics.
• The budget forces streaming/pipelining: partial transcripts into the LLM, LLM tokens into TTS, so audio plays at the first token — the demo’s 620 ms (usable) vs 1270 ms (broken) for waiting on the full response.
• Speech-to-speech realtime models (GPT Realtime, Gemini Live) collapse the cascade into one audio-in/audio-out step — lower latency, preserved prosody/emotion, native interruptions — versus the cascade’s modularity.
• Build it with Pipecat (pipeline orchestration) or LiveKit Agents (WebRTC transport + agent loop).

Quick check

Next

Voice agents are applied multimodal ML; see multimodal LLMs for the models and inference metrics for the TTFT/streaming that the latency budget depends on. For the framework landscape, compare agent runtimes and AutoGen.