Near-GPU TTS latency, zero GPU: what voice agents actually need in production

Versa 2.0 has produced first audio in 74 ms on one CPU thread.

In our RTX 4090 test, Kokoro's first cold synthesis took 2.44 seconds, with 14.9 seconds of model load time.

That is the attractive comparison. It is also incomplete.

Once warm, Kokoro generated a short phrase in 47 ms. It was faster than Versa, as a good GPU model should be. But that 47 ms was full-utterance generation, not streaming time to first audio. Our Kokoro setup did not stream. Versa's 74 ms figure is its fastest streaming time to first audio, while its production median is 139 ms.

These numbers are not an apples-to-apples model race. They expose the production decision teams actually face:

• optimize for the fastest warm benchmark on a GPU;
• or get interactive latency on ordinary CPU infrastructure, without making a GPU part of the serving path.

For many voice agents, the second option is more useful.

The benchmark, with the conditions left in

Versa figures are from live customer-facing infrastructure, serving calls at six flow-matching steps. Kokoro was run through our test harness on a spot RTX 4090 supplied through amics.ai, created by Alvaro Fragoso.

The comparison needs two warnings:

1. Versa reports streaming time to first audio. Kokoro reports the time to finish generating the full utterance because the tested setup did not stream.
2. The 74 ms and 2.44 s figures are deliberately the best Versa observation and the cold Kokoro observation. For steady-state throughput, warm Kokoro wins comfortably.

The point is not that a CPU beats a 4090. It does not. The point is that a voice product is more than its best warm inference number.

Cold starts are a product metric

A benchmark often assumes the model is loaded, the accelerator is allocated, kernels are warm, and traffic is steady. Production traffic does not always behave that way.

Voice agents are bursty. A campaign begins. A queue goes from zero to hundreds of calls. A worker is replaced. A region fails over. An autoscaler adds capacity. A low-traffic language has been idle.

In those moments, model load and first-request latency become user experience.

A 47 ms warm result can coexist with a 14.9 second load. Both are true. Only one appears in most benchmark headlines.

This changes how teams should test TTS:

cold path = provision + model load + first synthesis
warm path = request arrival + first audio + generation
recovery path = replacement worker + readiness + first successful request

Report all three. If a system scales to zero, cold latency belongs in the product SLO. If it never scales to zero, the cost of keeping it warm belongs in the infrastructure plan.

CPU inference changes that tradeoff. CPU capacity is easier to find, easier to autoscale, and does not require a separate accelerator pool. A team can keep latency low without reserving a GPU for every serving unit or designing around GPU availability.

Time to first audio matters more than time to finish

For a voice agent, users do not wait for the whole sentence to be synthesized. They hear the first chunk while the rest is still being produced.

That makes streaming time to first audio, or TTFA, the useful number. Full-utterance latency measures something else: how quickly a system can generate a completed file.

Both metrics matter, but for different products:

• Voice agents: TTFA, p90/p99 TTFA, chunk cadence, interruption behavior
• Audiobooks and batch generation: full-utterance latency and total throughput
• High-volume outbound systems: concurrency per node and recovery time
• On-device or private deployments: hardware requirements and memory footprint

A system that generates a sentence in 47 ms but cannot emit audio until the sentence is complete may still feel slower than a system that starts speaking in 139 ms and continues streaming.

That is why we report Versa's production median and tail, not only its best run:

• 74 ms fastest time to first audio
• 139 ms median
• 211 ms p90
• 0.26-0.36 real-time factor

At the complete voice-agent level, our measured caller-observed time to first audio is 385-442 ms in the best case and about 660 ms typically. The model number matters, but the caller experiences the whole pipeline.

What latency does a voice agent actually need?

Past a certain point, shaving another 20 or 30 ms from isolated TTS inference has less effect than fixing the rest of the turn.

The practical target is not "the lowest TTS number possible." It is:

1. Start audio quickly enough that the exchange feels responsive.
2. Keep p90 and p99 under control when traffic changes.
3. Stream continuously without audible gaps.
4. Leave room in the latency budget for turn detection, language-model output, transport, and telephony.
5. Recover without multi-second stalls when capacity changes.

Teams should benchmark the system in the state users will hit, not only a warmed-up notebook.

A useful test matrix looks like this:

Latency is not enough if speech quality breaks

Fast speech still has to be intelligible.

We use held-out text and transcribe synthesized output with the same recognizer used on genuine human recordings. This does not prove that synthetic speech is "better than humans." WER is a proxy for intelligibility, and a recognizer can have its own biases. It does give us a repeatable way to catch pronunciation regressions.

From the previous Versa version to Versa 2.0:

• English WER fell from 6.51% to 3.20%.
• Mean WER across ten benchmarks fell from 14.03% to 11.21%.
• On the held-out German, Italian, Catalan, and Basque sets, synthesized speech had lower WER than the genuine human comparator when both were scored by the same recognizer.

Each language set contained roughly 60 to 200 samples. We also measure an English UTMOS score of 3.14. Versa supports zero-shot voice cloning from an eight-second reference.

The important part is not one quality number. It is that latency work should be gated by repeatable quality checks. Otherwise a faster checkpoint may simply be speaking less clearly.

The infrastructure question behind the benchmark

GPU TTS can be extremely fast. Kokoro's 0.02 warm RTF on the 4090 makes that clear.

The tradeoff is operational dependency:

• accelerator availability by region;
• warm capacity during quiet periods;
• GPU-aware scheduling and autoscaling;
• recovery when a worker disappears;
• separate deployment paths for cloud, private VPC, edge, or on-prem environments.

Versa runs on x86 and Arm CPUs. In a measured AMD Genoa test, one node handled about 19 concurrent calls, equal to roughly $22 of infrastructure per concurrent stream per month. That test is not our current production deployment, so we treat it as a capacity result, not a production guarantee.

The hosted API follows the same idea. Public voice-agent rates range from $0.035 to $0.116 per minute depending on plan. For directional context, current public list rates for ElevenLabs and Deepgram are around $0.08 per minute. Feature bundles and billing units differ, so this is not a like-for-like price benchmark.

When CPU-native TTS is the better choice

Use a GPU model when maximum warm throughput is the main constraint and you already operate GPU infrastructure well.

CPU-native TTS is worth testing when:

• you want one deployment model across cloud and private infrastructure;
• traffic is bursty and cold capacity matters;
• GPU availability or regional coverage is a constraint;
• you need low-latency streaming without maintaining an accelerator fleet;
• voice is part of the product, but GPU operations should not become part of the company.

That is the problem we built Versa 2.0 to solve: near-GPU interactive latency, on a single CPU thread, in a production streaming API.

Try the live demo at lokutor.com, or get a free API key at app.lokutor.com.

If you benchmark it, test both cold and warm paths. Send us the numbers. We would rather compare real serving conditions than trade screenshots of ideal runs.