Every speech-to-text vendor claims to be the most accurate, and most of them have a benchmark that says so. Usually their own. Meanwhile, the field is racing to shave another 20 milliseconds off finalization, and every release gets announced as a win. If you are choosing a model for a voice agent, there’s a lot of different metrics.

Which is why independent benchmarks are so important.

Pipecat, maintained by Daily, runs an open-source one: 1,000 real utterances, streaming, scored for latency and semantic accuracy, with the code and the data published so anyone can rerun it.

The published repo matters for more than reproducing the numbers. You can read how the benchmark is built: which audio it uses, where the reference transcripts come from, what counts as an error. Those choices decide how much the results tell you about your own calls, which is why this gets you a shortlist and not a decision.

They ran the benchmark again around the launch of Linden, our Agent STT model purpose built for voice agents. It did not say we (Speechmatics) won.

What it said was more interesting.

Put latency on one axis and semantic accuracy on the other and you do not get a ranking. You get a curve, with several models sitting along it, each trading speed for meaning at a different rate.

That shape is called a Pareto frontier. A model sits on it if nothing else beats it on both speed and accuracy at once, and anything off it is beaten by something that is on it. The ones on the frontier are all defensible, and the benchmark cannot narrow it down any further than that.

So this post covers what the results say, and then the question that actually decides things. How much of it applies to you?

Another 150 milliseconds is buying you something. The question is whether you need it

TTFS is Time To Final Segment: the gap between the caller finishing speaking and the stable transcript your LLM can act on. The clock starts at the VAD's stop signal, which makes this a measurement of the whole pipeline. Swap the VAD, change the wait period (200ms in Pipecat) or let your STT close the turn itself, and every latency number here moves. Semantic WER asks whether the meaning survived rather than whether the words were identical, so it forgives “twenty five” for “25” and punishes losing the word “not”.

The September 2026 run left the frontier looking like this.

The other 16 are in the repo, beaten on both axes by something on this list. Vendors submit continuously, so treat any table in any blog post, including this one, as a snapshot.

Nemotron is the fastest model on the board, though Deepgram is the closer comparison if you are choosing between managed APIs. It returns a final segment about 120ms sooner than Linden, at one and a half times the semantic error rate. That does not make either better. It makes them different purchases, and the only useful question is whether your application needs what the 120ms buys.

A fast wrong answer makes the whole interaction slower

The failure worth planning around goes like this. A caller says “NE6 5AT” and the transcript arrives as “any six five eighty”.

Everything downstream then works as designed. The framework passes the string to the address lookup, the lookup finds no match, the agent asks the caller to repeat themselves. The prompt was fine, the tool call was correct given its input, and the interaction still failed. Whatever the STT layer saved on that turn is irrelevant next to the extra turn it caused. The fastest transcript is not the fastest completed interaction.

This is a specific class of error, not a rare one.

Standard WER weights all four the same. Semantic WER is a better proxy, and one reason this benchmark is more interesting than most. It scores wrong names, numbers and dropped negations as errors, and ignores punctuation, contractions and filler. But the underlying point stands on its own. A transcript can be overwhelmingly correct and still break the interaction, because a small number of tokens carry nearly all the consequences.

Which raises the obvious question. If the errors that matter depend on the application, how does a benchmark decide what counts?

Every benchmark is a set of choices. Here are this one’s

Pipecat makes its choices in public. That is the main reason to trust it, and it is also how you can see where it stops applying to you.

The audio is one corpus. The 1,000 utterances come from smart-turn-data-v3.1-train, collected to train a turn-detection model. Conversational speech, which puts it ahead of read-aloud corpora, but one collection with its own microphones and rooms. There is no telephony in it. Audio plays at real-time pace through a synthetic transport, so no 8kHz codec, no jitter, no packet loss, no barge-in.

The reference transcripts are model-generated. Ground truth comes from Gemini, then human review. That is careful practice, but it means the “correct” answer starts out as one model’s output so decisions on correction can be biased. And the places a reviewer is least likely to catch a mistake are names, codes and rare words, which are the tokens that carry the consequences.

“Does this error matter” is an LLM judgement. Semantic WER is scored by Claude against a published rubric. Number formats are ignored. Sensible default, though maybe not yours. If your agent reads order numbers back to the caller, “3” against “three” is getting a free pass.

Latency is conditional on a VAD. TTFS is measured from Silero VAD’s stop signal at a fixed threshold, with a 200ms wait window. Change the VAD or the endpointing and every figure above moves. This one catches people out, because the number looks like a property of the model when most of it belongs to the pipeline.

The tails are thin. A P99 drawn from 1,000 samples rests on about ten utterances. Rank on the medians. Treat the P99 column as directional: our own 690ms looks good against Meta's 1,922ms and poor against Deepgram's 326ms, and neither comparison rests on more than a handful of calls.

It is English, one speaker at a time. The big one, and easy to miss because the table looks so tidy. Every utterance is one person speaking English, nobody talking over them, no television in the next room. So the table says nothing about a caller with an unfamiliar accent, a caller switching language mid-sentence, or a line with a toddler on it. Nothing about speaker separation either. If your agent cannot tell who spoke, a partner saying “ask about the refund” in the background arrives in your LLM’s context as an instruction from the caller.

None of this is a criticism of Pipecat. In fact, the fact that you can interrogate the methodology is a strength. It’s more about what a benchmark like this is useful for: quickly narrowing 23 models down to a shortlist, without relying on vendor marketing.

But that’s probably where its usefulness ends. The five most accurate models here are within half a percentage point of each other, based on 1,000 utterances of clean English. That’s a pretty small margin to choose a supplier on, especially when performance can shift as soon as the audio gets harder or you move into other languages.

At that point, your own human-labelled calls, run through the same open-source framework, will tell you much more.

Why we built Linden to deliver accuracy over being the fastest

We were not trying to just minimize latency. We were aiming to get closer to the accuracy end of the frontier without letting latency become noticeable, to ensure voice agents reduce how often they ask people to repeat themselves.

Alphanumerics got their own treatment, because digits and letters read aloud do not follow the statistics of natural language. A model that has learned English well will happily "correct" a spoken sort code into something more probable and wrong. The prior that helps everywhere else actively hurts here, so phone numbers, account numbers and addresses needed targeted work rather than a general accuracy push. Short utterances are often another blind spot, these confirmations can be missed but that doesn't show up in WER metrics and that can lead to the agent not responding.

The bigger work is the part the benchmark cannot see. Linden runs the same 55+ languages as our realtime models, global-first, so one English pack covers British, American, Australian, New Zealand and non-native speakers rather than asking you to pick an accent in advance. That matters for a voice agent, because an inbound call does not tell you who is on the other end before it connects. We support transcribing callers who switch language mid-sentence, and Linden returns __speaker-attributed segme__nts rather than a word stream, so the LLM can follow a conversation even if there are multiple speakers talking to the agent.

None of that shows up in the results, and that absence matters as much for us as for anyone else. An English-only, one-speaker benchmark can't confirm any of the claims above. If your callers aren't all speaking one language, in a familiar accent, down a clean line, alone in a quiet room, these results tell you very little about any of the 23 models here, ours included.

Which is why you should test your own audio, on real calls.

What the benchmark cannot tell you

A benchmark shows how a model performs under test conditions. It cannot tell you what happens at peak traffic, whether the provider can support the regions you need, or what the experience is like when something breaks at 3am and you need real technical help.

Those are questions worth asking every provider on your shortlist, including us.

Some of the models above are very strong, but they are also backed by relatively new operations. That matters. Leaderboards can change in a matter of weeks; your integration is likely to be around for much longer.

What to do next

Use the benchmark to build the shortlist. Use your own calls to make the decision.

1. Run it yourself. The repo is open source and the dataset is on Hugging Face, so every number here is reproducible in five commands.
2. Then swap in your own recordings, at your own sample rate, through your own VAD settings, weighted the way your callers are actually distributed. If a fifth of your calls are in Spanish, or arrive over a bad line, or have two people on them, the sample should say so.
3. Score the errors that trigger actions separately from overall WER. That is where two models on the same point of the curve come apart.

If you want Linden in that comparison, the Pipecat quickstart will have it running in a pipeline in a few minutes, and the Agent STT docs cover LiveKit, Vapi and the raw WebSocket.