Design a Real-Time Voice AI

You're asked to design a voice assistant that holds a spoken conversation. A customer calls a support line, or taps a microphone button in an app, and talks to an AI that understands them, looks things up, and answers out loud.

Speech is less tolerant of latency than text chat. A pause longer than about a second feels broken. Let's build a latency budget first and use it to decide which stages must overlap.

Clarify the requirements

• What's the acceptable pause before the assistant starts speaking? Natural conversation turns over in 200 to 500 milliseconds. Anything past a second feels broken. This number governs the architecture.
• Can the caller interrupt? Yes in any usable product, and supporting interruption is a meaningful chunk of the design.
• How specialized is the vocabulary? Ask because a general model may mishear domain terms, requiring recognition biasing and domain-specific evaluation.
• Does it take actions or only answer? Taking actions on a caller's account means action tiers, idempotent tool calls, and approval gates, which are covered in depth in Design an AI-Powered Customer Support System.
• Phone or app? Telephone audio is narrowband and noisy, while app audio is cleaner. Confirm which input we need to support.

Assume: sub-500ms response, interruption supported, a specialized domain vocabulary, some actions, and telephone-quality audio.

Back-of-envelope numbers

• Concurrent calls: $10\text{k}$ at peak
• Audio bandwidth: $10\text{k} \times 32\text{ kbps} \approx 320\text{ Mbps}$ in each direction, which is small
• Latency budget for one turn: speech detection $100\text{ms}$ + final transcription $150\text{ms}$ + model first token $300\text{ms}$ + speech synthesis start $100\text{ms}$ $\approx 650\text{ms}$ before overlapping
• Turns: $10\text{k calls} \times 1 \text{ turn every } 10\text{ sec} = 1\text{k turns/sec}$ through the whole pipeline
• Cost per minute: transcription, model, and synthesis together run cents per minute, against a human agent at roughly $\$1$ per minute

Write down the latency budget first. Sequential stages already total about 650ms, so we need to overlap them to target a response below 500ms.

High-level architecture

Components

1. Media gateway. Terminates the audio connection, using WebRTC from an app or SIP from the phone network.
2. Turn detection. Decides when the caller has finished speaking, and when they've started again.
3. Streaming transcription. Emits partial text continuously rather than waiting for silence.
4. Orchestrator. Owns the turn, retrieval, tool calls, and cancellation.
5. Retrieval index. Domain knowledge, embedded for lookup.
6. Model. Generates the response, streamed token by token.
7. Streaming synthesis. Converts text to audio as it arrives, in chunks.

Deep dive 1: overlapping the pipeline

Run transcription, then reasoning, then synthesis in sequence and you get the 650 milliseconds computed above, plus whatever silence you waited through to decide the caller was done. Every stage has to start before the previous one finishes.

• Transcribe continuously. Streaming recognition emits partial hypotheses as the caller speaks, revising them as more audio arrives. By the time they stop, the transcript is essentially complete, and the final result is a confirmation rather than a computation.
• Start reasoning on the partial transcript. When turn detection is confident the caller is finishing, the orchestrator can begin retrieval and even a speculative model call on the partial text. If the last words change the meaning, cancel and restart, which costs a wasted call and saves hundreds of milliseconds on the majority of turns that don't change.
• Synthesize the first sentence before the last one exists. Text-to-speech does not need the whole response. We can send the first clause as soon as the model emits it, starting playback while generation continues.
• Retrieve in parallel with everything. The lookup runs against partial text at the same time as the rest, rather than as a step the model waits on.

The result is that the perceived latency is the time until the first syllable of audio, not the time until the response is complete, and the first syllable can arrive while the model is still writing the rest.

A short filler helps more than it should. A brief "let me check that" while a slow lookup runs makes a two-second wait feel conversational instead of broken. It's a small trick and it's what real deployments do.

Deep dive 2: turn-taking and interruption

Knowing when the caller has finished is a genuinely hard problem, and getting it wrong is the most common reason these systems feel bad.

Silence alone is a poor signal. A short threshold interrupts people who paused to think; a long one makes the assistant feel sluggish on every turn. The better approach combines voice activity detection with the content of the partial transcript, since a grammatically complete sentence with falling intonation is a much stronger end-of-turn signal than silence duration alone. Semantic turn detection is what modern systems use, and naming it distinguishes an informed answer.

Interruption, or barge-in, has to cancel everything downstream. When the caller starts speaking while the assistant is talking, three things must happen immediately: stop audio playback, cancel the in-flight model generation so you stop paying for tokens nobody will hear, and discard the pending synthesis.

The subtle part is what the assistant believes it said. It was interrupted at the third sentence of five, so the conversation history should record what was actually played, not what was generated. Otherwise the assistant proceeds as though it delivered information the caller never heard, and the conversation quietly desynchronizes. Tracking playback position and truncating the transcript there is a small detail that makes a large difference.

Echo cancellation is a prerequisite, not an optimization. Without it the assistant's own output arrives back through the microphone and it interrupts itself continuously.

Deep dive 3: embeddings for a specialized domain

A specialized domain uses embeddings in three different roles. Let's separate recognition biasing, retrieval embeddings, and speaker embeddings so we can evaluate and improve each one independently.

Recognition biasing. A general speech model has never seen your product names, and it will confidently transcribe them as common words that sound similar. Providing a domain vocabulary of product names, drug names, and the caller's own contacts and account labels lets the recognizer bias toward those tokens. Modern systems accept a phrase list per session, and the useful refinement is making it contextual: bias toward the terms plausible at this point in the conversation rather than the entire catalog, since a vocabulary of ten thousand terms dilutes the effect while fifty relevant ones sharpen it.

Retrieval embeddings, tuned for the domain. Off-the-shelf embedding models are trained on general text and don't know that two of your internal terms mean the same thing, or that two similar-sounding product tiers are entirely different. Two things help, in increasing order of effort:

• Fine-tune the embedding model on domain pairs, each a question with the passage that answers it, drawn from real transcripts. This is one of the highest-return interventions available, because it improves every retrieval the system will ever do.
• Train against hard negatives, meaning passages that look similar but are wrong. Teaching the model to separate the enterprise plan's cancellation policy from the individual plan's matters far more than teaching it to separate cancellation from billing, because the confusable pairs are what it actually gets wrong.

Speaker embeddings are a different thing entirely, mapping a voice rather than meaning. They support diarization, which is telling apart two people on the same line, and voice verification as one factor of identity. That second use needs care: voice can be cloned, so it belongs as one signal alongside others rather than as authentication on its own.

Evaluate them separately. Recognition is measured by word error rate on domain terms specifically, not overall. A system with 5% overall error that misses every product name is useless, and the aggregate number hides it. Retrieval is measured by recall@k on a golden set. Keeping those two numbers apart tells you which layer to fix, and that matters more in a voice system than a text one, because a recognition error corrupts every stage after it.

Deep dive 4: Pipeline vs. speech-to-speech

The pipeline, meaning separate recognition, model, and synthesis stages, is the architecture described above. Each stage is independently swappable, debuggable, and tunable; you can log the transcript, bias recognition, and change voices without touching anything else. The cost is accumulated latency across three hops, and the loss of everything that isn't words: tone, hesitation, emphasis, emotion all disappear at the transcription step.

Speech-to-speech models take audio in and emit audio out, with no text in between. They're markedly lower latency and preserve prosody, which makes them sound dramatically more natural. What you give up is exactly what the pipeline's seams provided: no transcript to log or audit, no place to inject domain vocabulary biasing, harder debugging, and no clean point to enforce a guardrail.

The choice should follow from the requirements rather than novelty. A regulated domain that needs transcripts and specialized vocabulary favors the pipeline; a consumer product where naturalness is the product favors speech-to-speech; and a hybrid can use speech-to-speech for conversation with a pipeline path for anything requiring a tool call or an audit record.

What breaks, in the order it will:

• Recognition on domain terms, which is why biasing and domain-specific evaluation come first.
• Turn detection, cutting people off or leaving dead air, which is the complaint users report most.
• A slow tool call blowing the latency budget, which is what fillers and aggressive timeouts exist for.
• Network jitter on mobile or telephone audio, needing a jitter buffer that trades a little latency for intelligibility.
• Cost under concurrency, since every one of ten thousand calls holds an open pipeline rather than making discrete requests.

Common pitfalls

• Running the stages in sequence. The budget is exceeded before you start; overlapping is the design.
• Waiting for the full response before synthesizing. Streaming the first clause is the largest single latency win available.
• Silence-only turn detection. It either interrupts thinking pauses or leaves dead air.
• Recording what was generated rather than what was heard. The assistant then believes it said things the caller never received.
• Using a general recognizer on specialized vocabulary. It mishears domain terms confidently, and every later stage inherits the error.

Leveling signals

Related lessons

Design an AI-Powered Customer Support SystemMedium

Answer customer questions from company data and take actions on their accounts.

Design ChatGPTHard

Serve a multi-turn conversational product backed by a large language model.