Voice AI: Speech-to-Text, TTS & Realtime Voice Agents

Voice AI

Voice AI covers systems that let people talk to software and be understood — voice assistants, phone agents, dictation, and real-time conversational interfaces powered by LLMs. The recent leap came from two directions: speech recognition and synthesis got good enough to feel natural, and LLMs became capable enough to hold a real conversation. Put them together and you get software you can talk to rather than type at.

The engineering challenge is latency and turn-taking. Text chat tolerates a second or two of thinking; a voice conversation does not. Humans expect a reply within a few hundred milliseconds, notice awkward pauses immediately, and interrupt each other constantly. Building voice AI that feels natural is largely about shaving milliseconds off a pipeline and handling the messy dynamics of real conversation. This page covers the architecture and the tradeoffs.

TL;DR

Voice AI has two architectures: the cascaded pipeline (STT → LLM → TTS) and speech-native models that take audio in and emit audio out directly.
The cascaded pipeline is flexible and debuggable — you can swap each stage and reuse your existing text LLM stack. Speech-native models are lower-latency and more expressive but less controllable.
Latency is the product. Natural conversation needs sub-second response; every stage (transcription, thinking, synthesis) eats the budget. Stream everything.
Turn-taking is hard: voice activity detection (VAD) decides when the user stopped talking, and barge-in lets them interrupt the agent mid-sentence.
Stream partial results — start the LLM on partial transcripts, start TTS on the first sentence — to hide latency.
Production voice adds real concerns: telephony integration, background noise, accents, cost per minute, and privacy of recorded audio.

Quick Example

The cascaded pipeline in its simplest streaming form — transcribe, think, speak, with each stage feeding the next as soon as it has partial output:

Synthesizing per sentence means the agent starts speaking before it has finished thinking — the single biggest perceived-latency win.

Core Concepts

The Two Architectures

The cascaded pipeline dominates production today because it's controllable — you can read the transcript, apply guardrails, call tools, and log everything. Speech-native models are advancing fast where latency and expressiveness matter most.

The Three Stages

Speech-to-text (STT / ASR) — converts audio to text, ideally streaming (emitting words as they're spoken) with interim and final results. Quality varies with accent, background noise, and domain vocabulary.
The LLM — your existing text model, doing the reasoning, tool calls, and retrieval. In voice, its output should be shorter and more conversational than a chat model's — nobody wants a bulleted essay read aloud.
Text-to-speech (TTS) — converts the reply to audio, again streaming so playback starts before the whole sentence is synthesized. Modern neural TTS is near-human; voice cloning and emotional control are common features.

The Latency Budget

Natural conversation has a response-time expectation of roughly 200–500 ms — the gap humans naturally leave between turns. A cascaded pipeline must fit transcription finalization, LLM time-to-first-token, and TTS time-to-first-audio into that window. Since the stages are sequential, the only way to hit it is to overlap them by streaming: begin LLM inference on a near-final transcript, and begin TTS on the LLM's first sentence. Waiting for each stage to fully complete before starting the next produces the robotic, laggy feel that kills voice UX.

Turn-Taking and Interruptions

The hardest part of voice isn't the AI — it's the conversational dynamics.

Voice Activity Detection (VAD)

VAD decides when the user has finished speaking so the agent can respond. Too eager and it cuts the user off mid-thought; too patient and there's an awkward pause. Good systems use semantic endpointing — combining silence detection with whether the sentence sounds complete — rather than a fixed silence timeout. This is one of the most-tuned parts of any voice agent.

Barge-In

Humans interrupt. If the agent is talking and the user starts speaking, the agent must stop immediately, discard the rest of its planned speech, and listen. Without barge-in, the agent talks over the user — instantly unnatural. Implementing it means monitoring the mic while speaking and being able to cancel in-flight TTS and LLM generation.

Backchannels and Filler

Natural conversation has "mm-hmm," brief pauses, and quick acknowledgments. Some voice agents emit filler ("let me check that…") to cover tool-call latency, keeping the conversation feeling alive while the backend works.

💡 Tip: Keep spoken replies short. A text model loves to enumerate; a voice agent that reads a five-bullet list aloud is exhausting. Prompt for one or two sentences, and offer detail only if asked.

Best Practices

Stream Every Stage

Streaming STT, streaming LLM tokens, and streaming TTS are non-negotiable for natural latency. Synthesize and play speech sentence-by-sentence as the LLM produces it. This overlap is what turns a 3-second pipeline into a sub-second feel.

Prompt for Spoken Style

A voice agent needs a different system prompt than a chatbot: short sentences, no markdown, no lists read aloud, natural spoken phrasing, and confirmation of what it heard when it matters. Numbers, dates, and spellings need care — "twenty twenty-six" reads better than "2026" through some TTS.

Handle the Real World

Design for background noise, cross-talk, accents, and mishearing. Confirm critical inputs ("Did you say Wednesday?"), let users correct the agent, and degrade gracefully when transcription is uncertain rather than confidently acting on a misheard command.

Keep Text Guardrails in the Loop

A key advantage of the cascaded pipeline: the transcript is text, so all your guardrails, moderation, and tool-permission logic apply. Don't bypass them just because the interface is voice — a voice agent that can take actions needs the same action-gating as any agent.

Watch Cost and Privacy

Voice is billed per minute across STT, LLM, and TTS — it adds up. And you're handling recorded human speech: get consent, minimize retention, and treat audio and transcripts as sensitive personal data under your privacy obligations.

Common Mistakes

Waiting for Each Stage to Finish

No Barge-In

An agent that can't be interrupted talks over users and feels broken. Monitor the mic during playback and cancel in-flight speech the moment the user speaks.

Reading a Chatbot Response Aloud

FAQ

Cascaded pipeline or speech-native — which should I build?

Start with the cascaded pipeline (STT → LLM → TTS) for almost any production voice agent: it reuses your text LLM stack, lets you inspect the transcript, apply guardrails, and call tools, and it's easy to debug. Reach for speech-native models when ultra-low latency and emotional expressiveness are the whole point and you can accept less control.

Why does my voice agent feel laggy even on a fast model?

Almost always because the stages run sequentially instead of overlapping. The fix is streaming end-to-end: emit partial transcripts, start the LLM before the transcript is fully final, and synthesize speech sentence-by-sentence so the agent starts talking while still thinking. Perceived latency is about time to first audio, not total pipeline time.

What is barge-in and why does it matter?

Barge-in is letting the user interrupt the agent mid-sentence — the agent stops speaking immediately and listens. Humans interrupt constantly, so without it the agent talks over people and feels unnatural. It requires monitoring the microphone during playback and being able to cancel in-flight TTS and LLM generation.

How do I make the agent stop talking at the right time?

That's turn detection, handled by voice activity detection (VAD). Fixed silence timeouts are crude; better systems combine silence with semantic endpointing — judging whether the sentence sounds complete — so the agent neither cuts users off nor leaves awkward pauses. It's one of the most-tuned parts of a voice system.

Does voice AI need special guardrails?

It needs the same guardrails as any LLM app, plus attention to spoken-context risks. With the cascaded pipeline the transcript is text, so your moderation, guardrails, and tool-permission logic apply directly — don't skip them because the interface is voice. Also treat recorded audio as sensitive personal data: consent, minimal retention, secure handling.