Every part of a speech→LLM pipeline is fast enough on its own. Put them in a row and you get three seconds, which is far too slow when a human is waiting for you to say something.
I build a desktop overlay that listens to the other side of a video call, transcribes it, and streams an answer. The budget I care about is time from the speaker finishing a sentence to the first token on screen. Here is where that second and a half went, and what actually moved it.
The naive pipeline
mic/loopback → PCM → WebSocket STT → final transcript
→ "is this a question?" classifier → LLM → stream
Roughly 3.2s to first token. Four places to attack, and only two of them turned out to matter.
- Stop sending silence The obvious win, and it is not about bandwidth — it is about the STT server's own endpointing. If you stream continuous audio, the server never sees a clean pause and delays its segment commit.
So VAD runs client-side in an AudioWorklet, on the raw signal before normalization:
this._vadThresh = 0.005; // RMS: above → speech
this._hangoverSec = 1.0; // keep sending after level drops
this._prerollMax = 14; // ~600ms @48k of buffered silence
Three parameters, and each one exists because of a specific failure:
Threshold on raw audio. I normalized first, and normalization amplifies room noise into "speech". Detect on the raw signal, normalize afterwards.
Hangover of 1.0s. Cutting the stream the instant RMS drops chops the tail off every sentence. It also has to exceed the server's own silence threshold (0.6s) with margin, or the server never gets the trailing silence it needs to commit the segment.
Pre-roll of ~600ms. A quiet sentence onset sits below the VAD threshold. By the time you detect speech, the first syllable is gone. So keep a rolling buffer of the last 14 chunks and flush it when VAD opens.
That last one is the difference between "what's a database index" and "at's a database index" — and the LLM answers the second one confidently and wrongly.
- Let the STT server do endpointing ElevenLabs Scribe v2 Realtime supports commit_strategy: 'vad', so the server commits a segment after a pause instead of waiting for you to ask:
commit_strategy: 'vad',
vad_silence_threshold_secs: '0.6',
Combined with client-side gating, the transcript arrives while the person is still drawing breath.
One sharp edge: keyterm biasing caps at 50. Send 51 and the socket closes with code 1008 and a message you will not see unless you log close reasons. I lost an evening to that.
- Reconnects, and the "muted socket" bug Long calls mean dropped sockets. Reconnecting is easy; reconnecting correctly is not.
The failure mode: a reconnect fires, and while it is in flight the user switches language, which triggers another reconnect. Now two sockets exist. The older one still delivers events, the newer one is the real connection, and the transcript interleaves garbage from both. Or worse, the stale one wins and the live one is silently ignored — audio flows, nothing appears, no error anywhere.
The fix is a generation counter:
const myGen = ++this.gen; // this connection
// ...later, in every handler:
if (myGen !== this.gen) return; // a newer connection superseded us
Every async continuation checks whether it is still the current generation. Anything from a superseded socket is dropped on the floor. It is four lines and it removed a whole class of "it just stops working after twenty minutes" reports.
- Deleting the classifier (the counterintuitive one) The pipeline had a cheap model deciding "is this a question worth answering?" before spending a call on the expensive one. It cost ~200ms and it seemed obviously correct.
It was the single worst component in the system.
Not because of latency — because of what it got wrong. Real conversation is full of follow-ups that are not questions in isolation:
"What is SOLID?"
— answer —
"And the second letter?"
"And the second letter?" scores as not-a-question. The classifier stayed silent exactly when context made the intent obvious. Precision was fine; the failures were catastrophic and clustered in the most valuable moments.
I deleted it. The main model now sees every completed utterance plus recent context and decides for itself — it has the context the classifier never had. That removed 200ms and fixed the follow-up problem. Two wins from deleting code.
What replaced it is much dumber and works better: a client-side completeness gate.
const AUTO_SILENCE_MS = 900; // no "?" — pause mid-sentence
const AUTO_UTTEREND_MS = 120; // ends with "?" — react almost immediately
If the buffer ends in a question mark, the sentence is over — fire in 120ms. Otherwise wait 900ms in case they are just thinking. No model call, no network hop.
- Prompt caching, and why the TTL matters more than the hit rate The system prompt carries a compressed CV card, so it is not small. Anthropic's prompt caching handles that — but the default TTL is 5 minutes, and interviews have pauses longer than 5 minutes. The interviewer talks, the candidate thinks, and the cache quietly expires. You then pay full price on exactly the questions that come after a long pause.
Two blocks, both at a 1-hour TTL:
const CACHE_1H = { type: "ephemeral", ttl: "1h" } as const;
// block 1: static persona — shared across every session and user
// block 2: session data (CV card, role) — stable for one interview
Splitting static from per-session matters: block 1 is identical for everyone, so it is warm before the user's first question. Get greedy and interpolate anything variable into it — the answer language, say — and you shatter one cache entry into sixteen.
- Parallelise the gatekeeping Auth, rate limit, quota check, body parse. All independent, all were sequential.
const [, accessRes] = await Promise.all([
enforceRateLimit(db, userId, "ask", 15, 60),
assertAccess(db, userId, "ask"),
assertDailyCap(db, userId),
req.json().then((b) => { body = b; }),
]);
~92ms, for free, with identical guarantees. Unglamorous and the best ratio of effort to result in the whole list.
- Pick the model on measurements, not vibes I assumed the frontier model was required. Measured across 10 technical questions:
cost / 500 requests full answer
Sonnet-class $3.54 8.6s
Haiku-class $1.03 5.2s
3.4× cheaper and 3.4 seconds faster. The price is roughly one factual slip every 5–6 questions and slightly rougher prose. For a real-time assistant where a late answer is worth zero, that trade is not close.
The enforcement point matters too: the client asks for a model, but the server owns the allow-list. Updates ship manually, so a client-side default would mean every already-installed copy keeps requesting the expensive model forever.
Where it landed
~1.1s from end-of-sentence to first token, from ~3.2s. The breakdown of what got it there is not what I expected going in:
deleting the classifier: −200ms and a correctness fix
VAD gating + server-side endpointing: −1.2s
parallel gatekeeping: −92ms
cheaper, faster model: −3.4s on full answer
The two biggest wins came from removing a component and from parameters in a 40-line audio worklet. Zero came from the LLM call itself, which is where I spent the first week.
What I would do differently
Log socket close codes from day one. The 1008 keyterm limit was invisible for a day because the close reason was never surfaced.
Treat "it silently stopped" as the default failure. Every quiet path — a superseded socket, an expired cache, a swallowed catch — looks identical to "working" from the outside. The generation counter and an idle watchdog on the stream exist because both failed silently first.
Measure before assuming the model is the bottleneck. It was 200ms of the 3.2s.
The overlay is Interview Copilot — Electron on the client, Supabase Edge Functions on the server so the STT and model keys never reach the desktop app. Happy to go deeper on any part of the pipeline in the comments.
