A scripted voice-agent demo works with one clean speaker. The first production call had crosstalk, and turn-taking fell apart. This is how I fixed it.
The demo went perfectly. It always does. One person, one microphone, a quiet room, and a script we had rehearsed maybe forty times. The agent listened, waited its turn, answered in about 800ms, and everyone in the room nodded. We shipped it to a pilot customer that Friday.
The following Monday, 9:14am, the first real call came in. A support line for a property-management company. The caller was in a car. Her husband was in the passenger seat. Their kid was in the back. Three humans, one phone, all talking at once, and my careful little agent sat there and did the worst possible thing: it started answering the kid.
Week 1: the demo lie
Our stack was ordinary. WebRTC brought audio in from the browser and the phone bridge, a voice-activity detector decided when someone was speaking, and when the VAD said "silence for 700ms" we treated that as end-of-turn and fired the transcript at the LLM.
That endpointing rule is the whole problem, and I did not see it for two days.
With one speaker, a 700ms silence gap almost always means "I finished my sentence, your turn." The rule works. It works in every demo you will ever give, because demos have one cooperative speaker who pauses politely.
Real calls do not pause politely. People talk over each other. They finish each other's sentences. A gap in speaker A is not a gap in the conversation, it is speaker B leaning in. My VAD saw energy, saw a dip, saw energy again, and interpreted the dip as a turn boundary. So the agent barged in on the mother mid-thought to answer a question the four-year-old had half-asked.
Week 1, later: reading the receipts
I pulled the raw audio for that 9:14am call and looked at it in Audacity like it owed me money. Then I ran our VAD offline, frame by frame, and logged every speech/no-speech flip with a timestamp.
Here is roughly what the first 6 seconds looked like once I lined it up:
0.00s speech (mother: "hi I'm calling about the")
1.42s speech (kid, overlapping: "MOM can we")
1.80s silence <- 240ms dip. NOT a turn end.
2.05s speech (mother continues: "about the deposit on")
3.10s silence <- 90ms. breath.
3.20s speech (father, low: "the Oakwood place")
4.60s silence <- 810ms. agent fires here. too late, wrong context.
The agent had already committed to a response at the 1.80s mark internally, buffered it, and then a second endpoint at 4.60s made it dump the whole thing. It answered "the deposit" question using audio that had three speakers braided together. The transcript it sent to the LLM was word salad, because our ASR was single-channel and had no idea two mouths were fighting for the same 8kHz of bandwidth.
Two problems, not one. First, I was detecting speech but not detecting who. Second, my endpointing logic assumed silence meant "conversation turn over" when it often just meant "this one speaker took a breath."
Week 2: VAD is necessary, not sufficient
First fix was the easy one. I had been using the VAD that shipped with WebRTC (the old GMM-based one). It is fast and it is fine for gross energy gating, but it flaps a lot on overlapped speech and car noise. I swapped the gate for Silero VAD, which is a small neural model and much steadier on noisy input.
One thing that bit me: Silero VAD (v4 and v5) wants exactly 512 samples per chunk at 16kHz. That is 32ms. Not 30, not 480 samples. If you feed it the wrong window it silently gives you garbage probabilities. Ask past-me how he knows.
import torch
import numpy as np
model, utils = torch.hub.load(
repo_or_dir="snakers4/silero-vad",
model="silero_vad",
trust_repo=True,
)
SAMPLE_RATE = 16000
CHUNK = 512 # Silero requires exactly this at 16kHz. 32ms.
def speech_probs(pcm_f32: np.ndarray):
"""Yield (t_seconds, prob) for each 32ms frame."""
for i in range(0, len(pcm_f32) - CHUNK, CHUNK):
frame = torch.from_numpy(pcm_f32[i : i + CHUNK])
prob = model(frame, SAMPLE_RATE).item()
yield (i / SAMPLE_RATE, prob)
Cleaner probabilities helped. The agent stopped triggering on tire noise. But it still could not tell the mother from the kid, so it still answered the wrong person. VAD tells you that someone is speaking. It never tells you who.
Week 2, the 11pm session: diarization
For "who," I reached for pyannote.audio. It does speaker diarization: given a chunk of audio, it returns time-stamped segments each labeled with a speaker id (SPEAKER_00, SPEAKER_01, and so on). It is not magic and it is not free (you run it as a heavier model, and on a live call you run it on a rolling window, not the whole call), but it was the piece I was missing.
from pyannote.audio import Pipeline
pipeline = Pipeline.from_pretrained(
"pyannote/speaker-diarization-3.1",
use_auth_token=HF_TOKEN,
)
# rolling window of the last ~8s of the call
diarization = pipeline({"waveform": window_tensor, "sample_rate": 16000})
for turn, _, speaker in diarization.itertracks(yield_label=True):
print(f"{turn.start:.2f}-{turn.end:.2f} {speaker}")
# 0.00-1.60 SPEAKER_00 (mother)
# 1.42-1.95 SPEAKER_01 (kid, overlaps SPEAKER_00)
# 3.20-4.55 SPEAKER_02 (father)
Now I could see the overlap explicitly. SPEAKER_01 starts at 1.42s while SPEAKER_00 is still going until 1.60s. That 180ms of true overlap is exactly what the naive endpointer had misread as a turn boundary.
Week 3: turn-taking that respects overlap
The real fix was not any single model. It was rewriting the endpointing logic to combine three signals instead of one:
- Is anyone speaking right now (Silero VAD probability over a short window).
- Who is the primary speaker (the diarization label with the most energy in the current window).
- Has the primary speaker actually yielded (silence from that specific speaker past a threshold, while no new speaker has taken the floor).
The rule that shipped, in plain words: only treat a gap as end-of-turn if the person we are tracking as the primary speaker has been silent for more than 600ms and no other speaker has started in that gap. If a new speaker starts, we do not barge in, we re-anchor to whoever now holds the floor and keep listening.
class TurnTaker:
def __init__(self, silence_ms=600):
self.silence_ms = silence_ms
self.primary = None
self.last_primary_speech_t = None
def update(self, t, speaking, primary_speaker):
if speaking and primary_speaker is not None:
if primary_speaker != self.primary:
# floor changed. someone new is talking. do NOT interrupt.
self.primary = primary_speaker
self.last_primary_speech_t = t
return "listening"
if self.last_primary_speech_t is None:
return "listening"
gap_ms = (t - self.last_primary_speech_t) * 1000
if gap_ms > self.silence_ms:
return "end_of_turn" # safe to respond now
return "listening"
It is not sophisticated. It is a state machine that refuses to speak until one specific human has clearly stopped and no one else has jumped in. That single change took the "agent talks over the caller" complaints from most calls in the pilot to roughly one in a hundred over the next two weeks on our deployment. Not zero. One in a hundred. Overlap is genuinely hard and I stopped pretending I would solve it completely.
What shipped, and what I would tell past me
What shipped: WebRTC for transport, Silero VAD as the fast speech gate, pyannote.audio for diarization on a rolling 8-second window, and a turn-taking state machine that anchors on the primary speaker and waits for a per-speaker 600ms silence before responding. Diarization runs slightly behind real time, so I let it correct the primary-speaker label a beat late rather than blocking on it. Good enough.
What I would tell the version of me giving that flawless Friday demo:
The demo is a lie you tell yourself. One clean speaker in a quiet room is not your product, it is your best case, and your best case will never call the support line. Real audio arrives with three people in a moving car and a codec that already mangled it.
Silence is not a turn. A dip in energy means one mouth paused, nothing more. Do not let your agent treat a breath as an invitation.
And measure the thing that actually hurts. I spent two days optimizing response latency (the 800ms everyone loved in the demo) when the real defect was that the agent was fast at answering the wrong person. Fast and wrong is worse than slow and right on a phone call. Slow the agent down until it is sure whose turn it is, then make it fast.
The 9:14am call is still in my logs. I keep it around. It is the most honest test case I have.
