Voice agents treat the transcript as ground truth. The speech recognizer often tells you it is not sure, and we were ignoring it.
TL;DR: When I went back through a month of "the agent did the wrong thing" incidents on our voice agent, close to a third of them started with a transcription the speech recognizer had already flagged as low confidence. The agent acted on it anyway, because nothing downstream of the recognizer looked at the confidence score. We started routing low-confidence turns to a one-line confirmation, and that class of failure mostly went away.
The transcript is not ground truth, and the recognizer knows it
A voice agent is a pipeline, and the first stage hands the rest of the system a string. Everything after that, intent classification, tool calls, the whole agent, treats that string as what the user said. But the recognizer almost always hands you a confidence score alongside the words, and a low score is the recognizer telling you, in advance, that it is guessing. We were dropping that score on the floor and treating a guess exactly like a certainty.
What the incidents actually looked like
The failures were rarely the dramatic kind. They were a caller saying an order number that came through with two digits wrong, or a yes that the recognizer scored as a coin flip because of background noise, and the agent confidently proceeding to act on the misheard version. To the dashboards everything looked fine: the agent did exactly what its input said. The input was wrong, and the one part of the system that suspected the input was wrong had already said so and been ignored.
The fix was a confirmation turn, not a better model
We did not swap recognizers or fine-tune anything. We put a threshold on the confidence score for turns that lead to a state change. Below it, the agent does not act, it reflects back what it heard and asks the caller to confirm. "I have order four-four-one-two, is that right." Above it, it proceeds as before. The cost is one extra turn on the uncertain calls, which is exactly the calls where an extra turn is worth it. The wrong-action incidents that traced back to a low-confidence transcript dropped by most of their volume.
Why this is easy to miss
The confidence score lives at the bottom of the stack and the failures show up at the top, several components away, so nobody connects them. The recognizer's own metrics looked healthy, word error rate was fine on average. Averages were never the problem. The problem was the specific turns where the recognizer was uncertain and we acted as if it were certain, and you only see those if you carry the confidence score forward to the moment of action and log it there.
The Remaining question
The threshold is the hard part and I do not have a principled way to set it. Too high and you confirm everything and the agent feels slow and patronizing. Too low and you let through the misses you were trying to catch. We set ours empirically, just below the confidence level where our incidents clustered, which is reactive in exactly the way I keep complaining about. If anyone has a calibrated way to choose a confirmation threshold per intent rather than one global number, that is the comment I want to read.
