Voice AI apps are built from three APIs: speech-to-text (STT) to hear, an LLM to decide, and text-to-speech (TTS) to answer. Newer realtime speech-to-speech models collapse that cascade into one socket. The cascade wins when you need transcripts, tool calls, and per-component cost control; end-to-end wins on latency and emotional nuance.
Below is the engineering that decides whether a voice feature feels alive or feels like a walkie-talkie: streaming STT without committing bad partials, TTS chunking that starts audio before the LLM finishes, barge-in that does not corrupt conversation state, and the latency budget to hold each stage to.
The voice stack: STT → LLM → TTS, and when end-to-end wins
The cascade is three network hops: STT converts audio to text, the LLM produces a reply, TTS renders that reply back to audio. Each hop is separately observable, swappable, and scalable, which is why it remains the default for production voice products. Its cost is latency accumulation and information loss: three round trips, and an STT boundary that discards prosody, emphasis, and speaker identity the moment audio becomes a string.
End-to-end speech-to-speech models take audio in and emit audio out, preserving paralinguistics and removing a hop, and they usually win the “does this feel human?” test on short turns. What you give up is control: you cannot schema-validate audio, inspect text before it is spoken, or swap one component when a vendor degrades at 2 a.m.
- Choose the cascade when you need a transcript (search, analytics, compliance), when the agent calls tools or retrieves from a knowledge base, when output needs guardrails, or when per-component cost control matters. Pair it with our guide to AI function calling and tool use.
- Choose end-to-end when turns are short, affect matters more than facts, and no text artifact is required.
- Run both when you need the realtime feel and a transcript: end-to-end in the live loop, plus a parallel STT pass writing text to storage.
STT: streaming vs batch transcription
Batch transcription takes a complete file and returns one transcript. Streaming transcription takes frames as they are captured and returns partial hypotheses followed by a final when it decides the utterance ended. Voice agents need streaming; archives, podcasts, and uploads are better served by batch.
| Dimension | Batch (file) | Streaming (socket) |
|---|---|---|
| Input unit | Whole file | 20–100 ms audio frames |
| Output | One final transcript | Partials + final per utterance |
| Time to first text | Seconds to minutes | 200–400 ms after speech starts |
| Right context | Full — sees the whole file | Limited — a few hundred ms ahead |
| Accuracy on names | Higher | Lower without a keyword boost |
| Endpointing | Not applicable | Your responsibility |
| Use for | Uploads, archives, batch analytics | Live agents, captions, voice commands |
Three rules keep streaming STT from wrecking your turn logic. Never commit a partial: the trailing two or three words routinely change as more audio arrives, so a partial is a UI update, not a fact, and only the final drives your LLM call. Endpointing is a policy you own: silence-only endpointing fires mid-thought on “I’d like to book a flight to… um… Berlin,” so require trailing silence and a minimum utterance duration. Feed the model domain vocabulary: streaming models see less right context, so a keyword boost list of product names and proper nouns recovers most of the accuracy gap against batch.
import asyncio, json, websockets
# Streaming STT over a WebSocket. Send 100 ms frames, not 20 ms: tiny frames
# multiply per-message overhead and starve the model of context per inference.
WS_URL = "wss://your-gateway.example/v1/audio/transcriptions/stream"
SAMPLE_RATE = 16000
SILENCE_MS = 500 # endpoint after 500 ms of trailing silence
MIN_SPEECH = 250 # ignore bursts shorter than this (coughs, clicks)
async def transcribe(mic_queue, on_partial, on_final, api_key):
async with websockets.connect(
WS_URL,
extra_headers={"Authorization": f"Bearer {api_key}"},
ping_interval=20, # proxies kill idle sockets at ~30-60 s
max_size=None,
) as ws:
await ws.send(json.dumps({
"type": "session.start",
"encoding": "pcm_s16le",
"sample_rate": SAMPLE_RATE,
"interim_results": True, # ask for partial hypotheses
"endpointing": {"silence_ms": SILENCE_MS, "min_speech_ms": MIN_SPEECH},
"keywords": ["Kubernetes", "qoraapi", "Postgres"], # domain boost
}))
async def pump(): # mic -> socket
async for frame in mic_queue:
await ws.send(frame)
async def drain(): # socket -> callbacks
async for raw in ws:
msg = json.loads(raw)
if msg["type"] == "partial":
on_partial(msg["text"]) # UI only. never commit this.
elif msg["type"] == "final":
on_final(msg["text"]) # this is what your LLM sees
elif msg["type"] == "error":
raise RuntimeError(msg["message"])
await asyncio.gather(pump(), drain())TTS: streaming synthesis, voice consistency, and chunked synthesis
For TTS, the metric that matters is not total synthesis time but time to first audio chunk. A voice that starts speaking in 200 ms and streams slightly faster than real time feels instantaneous. A voice that renders the whole paragraph in 300 ms but starts at 900 ms feels broken. Optimize the first chunk, then keep the buffer ahead of playback.
That is why chunked synthesis matters: if you wait for the LLM to finish its whole reply before calling TTS, you pay full generation time plus TTS startup before the user hears anything. Pipelining speakable segments into TTS overlaps the two stages and removes most of the LLM’s generation time from perceived latency.
import re
# Speak the first clause while the rest of the answer is still generating.
# Split guard: do not break on decimals (3.5) or abbreviations (Dr., etc.).
BOUNDARY = re.compile(r"(?<!\b(?:Dr|Mr|Ms|St|vs|etc))(?<!\d)[.!?](?=\s|$)")
async def speak(llm_stream, tts):
buf, started = "", False
async for delta in llm_stream:
buf += delta
# Start on the first clause (~4-6 words) instead of the first sentence.
if not started and len(buf.split()) >= 4:
await tts.send(buf); buf = ""; started = True
elif BOUNDARY.search(buf):
await tts.send(buf); buf = "" # sentence boundary
if buf.strip():
await tts.send(buf)
await tts.flush() # signal end of utteranceTwo rules for chunk boundaries. Start on the first clause, not the first full sentence — waiting for a period adds hundreds of milliseconds for no gain. And never split mid-number or mid-abbreviation: a naive split on . turns “3.5 seconds” into “three” + “five seconds” and “Dr. Chen” into two utterances with an audible restart.
Voice consistency is an operational problem, not a modeling one. Pin the voice ID and the model revision, because providers update models in place and a silent revision bump changes timbre. Synthesize a turn’s chunks sequentially with identical settings — parallel synthesis plus concatenation produces audible seams — and when the provider exposes previous_text / next_text, pass neighboring text so prosody carries across the boundary. Add a weekly regression that synthesizes a fixed sentence and compares speaker embeddings to a stored baseline.
Realtime speech-to-speech: transport and barge-in
Realtime APIs run over WebSocket or WebRTC, and the choice is network physics, not preference. Start on WebSocket: one endpoint, and your server-side pipeline is identical either way. Move the client transport to WebRTC when you ship a mobile app on cellular networks, where 1% packet loss under TCP becomes audible stutter.
| Property | WebSocket | WebRTC |
|---|---|---|
| Transport | TCP | UDP (SRTP / SCTP) |
| Packet loss | Head-of-line blocking stalls all later audio | Concealment and FEC degrade gracefully |
| Echo cancellation | You implement it | Built-in AEC, AGC, noise suppression |
| Jitter buffer | You build it | Built-in and adaptive |
| NAT traversal | Simple | Needs ICE / STUN / TURN |
| Ops complexity | Low | High |
| Best for | Server-to-server, desktop, prototypes | Consumer mobile, phone calls, lossy networks |
Barge-in is the hard part, and the hard part is not detection — it is invalidation. When the user speaks mid-reply you must cancel server-side generation, stop local playback, and discard every audio chunk still in flight from the cancelled turn. Use a monotonically increasing epoch per turn: any chunk tagged with a stale epoch is dropped, no matter when it arrives.
# Barge-in with an epoch counter. Stale audio is dropped, not queued.
class VoiceTurn:
def __init__(self, ws, tts, history):
self.ws, self.tts, self.history = ws, tts, history
self.epoch, self.playing, self.played_words = 0, False, 0
async def on_user_speech_start(self):
self.epoch += 1 # invalidate everything older
if self.playing:
await self.tts.cancel() # stop playback now
await self.ws.send('{"type":"response.cancel"}') # stop generation
self.playing = False
# Critical: record only what the user actually HEARD.
# Without this the model "remembers" a sentence that was never played.
self.history.truncate_assistant(self.played_words)
async def on_tts_chunk(self, epoch, audio, words):
if epoch != self.epoch:
return # audio from a cancelled turn
self.playing = True
await self.play(audio)
self.played_words += words # track the playback watermark
async def on_tts_end(self, epoch):
if epoch == self.epoch:
self.playing = FalseThe bug that bites teams here is conversation state, not audio. If you append the assistant’s full reply to history when barge-in cut it off after four words, the model believes it said things the user never heard. Track a playback watermark and truncate the assistant message to the words actually played.
The other classic failure is the agent interrupting itself: without acoustic echo cancellation the microphone hears the speaker, VAD classifies it as user speech, and your barge-in handler cancels the reply it just started — an infinite loop. Fix it on the client with getUserMedia({audio: {echoCancellation: true}}) or WebRTC’s AEC, and require 150–250 ms of speech above the current playback level before accepting an interruption.
The latency budget that makes conversation feel natural
Conversation has a hard perceptual clock. Below roughly one second of silence after the user stops talking, the exchange feels responsive. Past about two seconds, users assume the agent is broken and start talking over it, which triggers barge-in and makes everything worse. Hold each stage to a budget rather than optimizing what is easiest to measure.
| Stage | Target (p50) | Ceiling (p95) | What it buys you |
|---|---|---|---|
| VAD speech detection | 20–40 ms | 60 ms | Fast barge-in without false triggers |
| Endpointing silence | 300–400 ms | 600 ms | The largest single lever on perceived gap |
| STT finalization after endpoint | 50–150 ms | 250 ms | Time from “done speaking” to text in hand |
| LLM time to first token | 200–400 ms | 700 ms | Start of the reply text |
| TTS time to first audio chunk | 150–300 ms | 500 ms | Time from text to audible speech |
| Network + jitter buffer | 30–60 ms | 120 ms | Transport and playout smoothing |
| Total perceived gap | ~800–1,300 ms | < 2,000 ms | Above ~2 s users start interrupting |
Two non-obvious consequences follow. TTS first-chunk beats LLM TTFT. Users forgive a slow answer far more than a late start, so a 300 ms TTS start with a 600 ms TTFT feels better than a 900 ms TTS start with a 300 ms TTFT — similar totals, but the first speaks sooner. Endpointing is a dial you can trade. Cutting trailing silence from 600 ms to 250 ms nearly halves the perceived gap but sharply raises mid-thought cutoffs. To get both, use semantic endpointing — a model that predicts turn completion from the partial transcript — so you cut fast on “what’s the weather” and wait on “I need to cancel my… actually, change it.”
Instrument every boundary with timestamps (speech_end → stt_final → llm_ttft → tts_first_chunk → playback_start) and log the derived response gap per turn. Aggregate p95, not averages — users experience the tail. If you already stream tokens to a browser, see our AI API streaming guide for the proxy-buffering trap that silently adds hundreds of milliseconds per chunk.
Cost and scaling: per-minute vs per-token
Voice has a different cost shape from text. STT and TTS are metered by audio duration (or characters for some TTS models); the LLM is metered by tokens. One minute of speech is roughly 130–160 words — only a couple hundred tokens of text. Audio therefore dominates the bill, so the levers that matter are the ones that reduce audio-minutes.
- You pay for silence. A duration-metered STT call receiving five minutes of audio bills five minutes, including three minutes of room tone. Gate the mic with VAD before the socket and stop sending when the user is silent — but use a ~200 ms hangover and a low threshold (around −40 dBFS), or you will clip word onsets for a marginal saving.
- Shorten the model’s answers. TTS bills output only, so reply length is the biggest TTS lever. Instruct the LLM for voice explicitly: one to three sentences, no markdown, no lists, no emoji. Capping replies at two sentences instead of six typically cuts TTS duration by well over half, and cuts LLM output tokens at the same time.
- Cache fixed prompts. Pre-rendered greetings and menu options cost nothing to serve, eliminate the latency and per-character charge on your most-played audio, and waste nothing when a user barges in.
- Accept barge-in waste. Some providers still bill audio you cancelled mid-stream. Keep replies short so the wasted fraction stays small rather than trying to eliminate it.
Scaling voice is a concurrency problem, not a throughput problem. Text APIs scale in requests per second; voice scales in sessions in flight. A three-minute average call at 1,000 concurrent sessions is only about 330 calls per minute — your ceiling is the provider’s concurrent-stream limit, not its RPM.
Bandwidth is the constraint people forget. 16 kHz mono PCM16 is 32 KB/s — about 256 kbps per direction, per call. At 1,000 concurrent calls your relay pushes roughly 32 MB/s of raw audio each way, and that egress is a real line item. Encode with Opus (~24 kbps) at the edge and decode only where a provider demands PCM: roughly a 10× reduction for a codec nobody notices on speech. Duration-gating, reply capping, and edge compression are most of a reduce AI API costs strategy for voice.
One API for STT, TTS, and the LLM
Three vendors means three keys, three auth schemes, three bills, and three places to look when latency regresses. An OpenAI-compatible relay collapses that into one base URL and one token: /v1/audio/transcriptions for STT, /v1/audio/speech for TTS, /v1/chat/completions for the LLM. Four benefits are concrete for voice:
- Regional co-location. Your cascade makes three round trips per turn. If STT, LLM, and TTS live in three regions, network RTT enters your latency budget three times. One gateway keeps the hops in one region.
- Mid-session failover. If a TTS stream drops, reopen against a second provider and continue the turn by changing a model string — no pipeline rewrite.
- One spend view. Per-minute audio cost next to per-token LLM cost is the only way to know whether STT or TTS dominates your bill.
- Free model swapping. When a cheaper STT model ships, you evaluate it by changing one parameter, not by rebuilding your audio path — the same argument that applies to multimodal AI APIs generally.
To test this without wiring three accounts, qoraapi.com exposes STT, TTS, and chat models behind one OpenAI-compatible endpoint, so the code above points at a single host with a single key.
Frequently asked questions
Do I need WebRTC, or is WebSocket enough?
WebSocket is enough for server-to-server pipelines, desktop apps, and prototypes, and it is far simpler to operate. Choose WebRTC for consumer mobile apps on lossy networks or when you need built-in echo cancellation and an adaptive jitter buffer: over TCP, one lost packet stalls every later packet, so 1% loss becomes audible stutter.
Why does my voice agent keep interrupting itself?
Missing acoustic echo cancellation. The microphone hears the speaker output, VAD labels it as user speech, and your barge-in logic cancels the reply it just began — repeatedly. Fix it at the client with echoCancellation: true from getUserMedia or WebRTC’s AEC, and require 150–250 ms of speech above the current playback energy before accepting an interruption.
Should I replace STT + LLM + TTS with one end-to-end speech model?
Only if you do not need text artifacts. End-to-end models cut a hop, preserve tone, and usually feel more natural on short turns, but you lose the transcript, deterministic tool-calling and RAG guardrails, and independent component swapping. If you need transcripts for search, analytics, or compliance, keep the cascade — or run end-to-end live with a parallel STT pass writing text to storage.
How do I keep the same voice across many TTS chunks?
Pin the voice ID and the model revision, synthesize a turn’s chunks sequentially rather than in parallel, and pass neighboring text through previous_text / next_text when the provider supports it. Add a weekly regression that synthesizes a fixed sentence and compares speaker embeddings to a baseline, which catches a silent model update before your users do.
Conclusion
Build voice in stages. Get streaming STT working with partials for UI only and finals driving your LLM turn. Pipeline TTS on the first clause so audio starts before generation ends. Implement barge-in with an epoch counter and a playback watermark so cancelled turns cannot poison history. Then hold every stage to the budget table, gate silence to stop paying for room tone, and cap reply length to control TTS minutes and LLM tokens.
Do those five things and your agent will feel responsive rather than robotic — and keep STT, TTS, and chat models behind one OpenAI-compatible endpoint so swapping any of them costs a parameter change, not a refactor.
Leave a Reply