Real-Time Voice API Latency: Why Deepgram, ElevenLabs, and Cartesia Numbers Can't Be Compared

[A waterfall chart: four stacked bars labeled ASR, LLM, TTS, Network. A small red bracket spans only the TTS bar, tagged '"time-to-first-byte: 150ms" — the number in the pitch deck.']

A vendor claims 150 milliseconds. Another claims 200. A third says its API is the fastest in the category. None of these numbers are lying, and none of them are measuring the same thing. That's the actual problem with real-time voice API latency marketing: it's not dishonest, it's incomparable.

Building a voice agent means stitching together speech recognition, language generation, and speech synthesis, sometimes across three vendors, sometimes inside one socket. Each stage has its own clock. Each vendor starts and stops that clock wherever makes their number look best. This piece is about learning to read those clocks correctly before you sign a contract based on a slide.

What Does 'Sub-300ms Latency' Actually Mean in Voice AI Marketing?

It usually means one pipeline stage, isolated and measured under ideal conditions, presented as if it represents the whole conversation. A '150ms latency' claim from a text-to-speech vendor is almost always time-to-first-audio-byte on a single stage, not the time from when a user stops talking to when they hear a reply.

The Number Everyone Quotes

Every voice AI landing page has a latency badge near the top. It's a real number, measured under real conditions, in a lab. The implication, though, is that a finished voice agent built on that API will feel that fast in production. It won't, because the number never included the other three stages of the pipeline.

What Gets Measured vs. What Gets Sold

What gets measured: one isolated segment, often the fastest one, on the shortest input the vendor could justify. What gets sold: the implicit promise that this number is what your users will experience end to end. The distance between those two things is where procurement decisions go wrong.

A latency number without a stage label is a marketing asset, not an engineering spec.

This sets up the core tension of building on any of these platforms: four vendors, four different stopwatch conventions, and no shared unit of comparison unless you build one yourself.

Why Aren't Deepgram, ElevenLabs, Cartesia, and OpenAI Realtime Numbers Comparable?

They aren't comparable because each vendor draws the measurement boundary around a different part of the pipeline, and none of the published numbers account for the stages happening outside their own product. Comparing them directly is like comparing a car's 0-60 time to a plane's cruising speed.

Where Each Vendor Draws the Stopwatch Line

Eleven Labs, scored 8.3/10 by the TopReviewed AI panel, typically benchmarks time-to-first-audio-byte on short, often cached or pre-warmed text strings, not on live token-by-token output streamed from an LLM mid-generation. That's a meaningfully easier problem than synthesizing speech from text that's still being written.

Deepgram's published figures tend to describe streaming transcription speed on the ASR side, how quickly partial text appears as someone talks. That's a different stage entirely from generating a response or speaking it aloud, and a fast ASR number tells you nothing about what happens after the transcript is ready.

OpenAI's Realtime API collapses recognition, reasoning, and synthesis into a single socket connection. That architecture changes what 'latency' even measures, since there's no clean seam between stages to isolate and market separately. A single round-trip number from a unified API is answering a different question than a stage-isolated number from a stitched vendor.

Cartesia markets model inference speed on GPU-optimized inference paths, numbers that reflect a controlled compute environment, not the public internet with real routing and contention.

ASR vs. TTS vs. Full-Duplex Claims

None of these headline figures include LLM time-to-first-token, the jitter introduced by network hops, or the delay from endpointing, the system deciding a user has actually finished speaking. In a real conversation, those three unmeasured factors often dominate what a person perceives as lag, more than any single vendor's optimized stage.

What Actually Makes Up End-to-End Voice Agent Latency?

End-to-end latency is the sum of four sequential stages, each with its own distribution: speech-to-text, language model reasoning, text-to-speech, and network transport, plus a hidden fifth factor, endpointing, that decides when the clock even starts.

The Four Stages Nobody Adds Together

• ASR (speech-to-text): converting audio to text, streaming partial results as the user talks
• LLM (reasoning/generation): time-to-first-token plus generation speed for the response
• TTS (text-to-speech): converting generated text back into audio, ideally incrementally
• Network transport: the round trips between client, ASR, LLM, TTS, and back, each with its own hop

Vendors publish numbers for their own stage. Almost nobody publishes the sum, because the sum depends on choices the vendor doesn't control: which LLM you pick, where your users are, how you handle endpointing.

Where Jitter Hides

Endpointing, deciding a user has stopped talking and it's the agent's turn, adds a delay that's rarely disclosed anywhere in vendor documentation. Too aggressive and the agent interrupts. Too conservative and the agent feels slow to respond, even if every other stage is fast.

Network jitter and geographic routing can swing real-world latency by more than the difference between two competing models. A GPU-optimized inference path in one data center means little if your users are three time zones away and routing through congested public internet.

[Waterfall diagram: ASR bar (short), LLM bar (long, variable width to show token-by-token variance), TTS bar (medium), Network bar (thin but present at every seam between the others). A single highlighted segment, tagged "marketed number," covers about a quarter of the total width.]

How Should You Benchmark a Voice Stack Before Choosing One for Production?

Benchmark the full round trip, not the isolated stage: measure from the moment a user stops speaking to the moment agent audio starts playing, under real network conditions, and report p50/p90/p99, not a single average.

A Standardized Measurement Framework

An average hides the tail, and tail latency is what breaks trust in a voice agent. A p50 of 400ms sounds fine. A p99 of 2.5 seconds means one in a hundred conversations feels broken, and users remember the broken ones. Test conditions matter as much as the metric: not localhost, not a same-region GPU cluster, but the actual conditions your users will connect from.

What to Log and Why

Use real conversational turns for testing, not isolated static strings fed straight into a TTS endpoint. Log each pipeline stage separately, timestamped, so vendors can be compared segment by segment rather than trusting a single bundled number from a slide deck.

Run the same test script across multiple geographic regions and connection qualities, WiFi, LTE, congested public networks, because a stack that looks identical on a fiber connection in one city can diverge sharply somewhere else.

• Instrument every stage boundary with a timestamp, not just start and end
• Use observability tooling like Honeycomb (8.5/10) or Grafana (8.5/10) to trace and visualize latency distributions across the pipeline, rather than trusting an averaged headline number
• Adapt Promptfoo-style eval workflows, normally used for scoring LLM output quality, to also score latency alongside correctness for each conversational turn

Which Pipeline Choices Change the Latency Math the Most?

The single biggest architectural lever is whether you stitch separate ASR, LLM, and TTS vendors together or use a unified realtime API that keeps everything in one socket. Stitched stacks add network hops. Unified APIs remove hops but remove choice.

Stitched Stacks vs. Unified APIs

A stitched architecture gives you control over model selection, swapping in Anthropic Claude API (8.3/10) for reasoning, or an open model like Llama (8.7/10) hosted via Hugging Face (8.9/10) for cost or customization reasons. Each swap is a network hop, and each hop adds latency variance that a vendor's own benchmark can't account for because it's outside their product boundary.

A unified API, like OpenAI's Realtime offering, collapses those hops into one connection. That's a real latency advantage on paper. The tradeoff is lock-in: one vendor's model quality, one vendor's pricing, one vendor's roadmap for improving any single stage.

Model Size and Streaming Tradeoffs

Streaming partial LLM output into TTS before the full response finishes generating can cut perceived latency substantially, since the agent starts speaking the first sentence while the model is still reasoning about the third. This only works if the TTS vendor's API actually accepts incremental text input rather than requiring a complete string. Check this specifically before assuming a stitched stack will stream well end to end.

How Do You Build a Latency Comparison Table That's Actually Fair?

A fair comparison table separates what a vendor publishes from what that number actually measures, and explicitly flags what's missing, rather than lining up four headline figures as if they were interchangeable.

Columns That Matter

The columns that matter are: the vendor, what they publish, what that figure actually measures, what's missing from the number, and a qualitative full round-trip estimate. Fabricated precision here is worse than no number at all, so anything not independently verified should be marked as coming from vendor documentation, not treated as measured fact.

A Sample Comparison

Vendor	What They Publish	What It Actually Measures	Missing From the Number	Full Round-Trip Estimate
Deepgram	Streaming transcription speed (per vendor docs)	ASR partial transcript latency only	LLM time-to-first-token, TTS, network, endpointing	Not disclosed; depends entirely on downstream stack
Eleven Labs	Time-to-first-audio-byte (per vendor docs)	TTS synthesis start time on short/cached text	Live token-by-token input handling, network, ASR, LLM	Higher when fed live streaming LLM output vs. static text
Cartesia	Model inference speed (per vendor docs)	GPU-optimized inference path performance	Public network conditions, geographic distance, jitter	Varies significantly outside controlled compute environments
OpenAI Realtime API	Unified socket round-trip figures (per vendor docs)	Combined ASR + reasoning + TTS in one connection	Comparable stage-by-stage breakdown; harder to isolate bottlenecks	Lower hop count, but tail latency still undisclosed at p99

Notice the last column never has a hard number. That's intentional. Nobody outside these vendors' own infrastructure teams can verify a true p99 without running the test themselves, which is exactly the point of the next section.

What Should Buyers Demand From Vendors Before Signing a Voice AI Contract?

Buyers should demand a stage-by-stage disclosure of what any published number measures, the network conditions it was measured under, and p90/p99 figures, not just an average, before any contract gets signed.

Questions to Ask in the Sales Call

• Which exact pipeline stage does this latency number cover?
• What network conditions and geographic distance were used during measurement?
• Can you provide p90 and p99 figures, not just a median or average?
• Can we see a live demo over a realistic connection, not a recorded clip?

Red Flags in a Benchmark Deck

A benchmark that only shows TTS time-to-first-byte on short static strings, with no mention of live streaming input, is a signal that the vendor hasn't stress-tested the part of the pipeline that matters most in production. Averages with no tail latency disclosed are another.

Reproducible test environments matter more than a single sales-cycle demo. Infrastructure-as-code tools like HashiCorp Terraform (8.6/10) let a team spin up identical test conditions across multiple vendor trials, so comparisons aren't skewed by inconsistent environments. Pairing that with edge testing through Cloudflare (8.3/10) helps control for network variance across regions, isolating the vendor's actual performance from the internet's mood that day.

Before the next vendor call, write down the single number from their pitch deck and ask them, on the record, which of the four stages it covers. If they can't answer immediately, that's the actual benchmark result.