Why I run speech-to-text locally instead of calling a cloud API
Yesterday I wrote about deploying gemma, bge-m3, and whisper on a single server without enough VRAM for all three. This post is about why whisper is one of those three.
The problem with cloud STT
When you call a cloud speech-to-text API—OpenAI's Whisper endpoint, Google Speech-to-Text, Amazon Transcribe—the audio travels to their servers. For most use cases that's fine. For mine it isn't.
The project I'm building transcribes work calls. Work calls contain client names, project specifics, sometimes pricing discussions. Sending that audio to a vendor's inference endpoint means it travels over the network and gets processed on hardware I don't control.
Running whisper locally means the audio file stays on the machine.
What the setup actually looks like
- Local server: RTX 3060 (12 GB VRAM)
-
faster-whisperinint8quantized mode - Audio files uploaded from a phone via a PWA, stored on the same machine, processed there
A transcription call looks like this:
from faster_whisper import WhisperModel
model = WhisperModel("medium", device="cuda", compute_type="int8")
segments, info = model.transcribe("recording.m4a", beam_size=5)
for segment in segments:
print(f"[{segment.start:.1f}s] {segment.text}")
No API call. No auth header. No request log on a vendor's side. The audio file doesn't leave the machine.
The VRAM situation
The 12 GB card is shared with two other models. Whisper medium in int8 uses roughly 2.5–3 GB of VRAM when warm. That's workable as long as the models don't run at the same time.
The three models load sequentially in my pipeline: transcribe → embed → analyze. Whisper runs first, which means it has the most headroom before the other two have loaded. When gemma is handling a VLM task and the card is under pressure, whisper drops to CPU. Slower—30–60 seconds per minute of audio instead of 8–12—but it completes without crashing.
I didn't anticipate needing that fallback. It showed up during the first few real recordings when I hadn't fully worked out the load order yet.
What this doesn't solve
Local STT doesn't fix transcription accuracy automatically. Whisper medium is good but not perfect. Names, domain-specific terms, and cross-talk all degrade it. I haven't run it on enough actual work calls to have a reliable accuracy number. On clean audio, 90%+. On a speakerphone with background noise, noticeably worse.
It also doesn't solve the downstream pipeline. Right now I can transcribe a call and get a text file. The next piece—extracting structured information from that transcript, linking it to what came in over text and email—is what I'm working on now. That part isn't done.
Why not a hosted Whisper endpoint?
Hosted Whisper endpoints exist and some are cheap. If data residency doesn't matter, they're probably the right call: faster, no hardware to manage, no VRAM budgeting.
The asymmetry is the point. A slower local run that keeps audio in-house is worth the tradeoff when the content of the calls is the thing you're trying to protect. The cost of audio leaving is not symmetric with the cost of running it locally.
Where this fits
I'm building a phone assistant that collects what comes in over calls, texts, and emails—and turns it into something useful instead of leaving it scattered across three different apps. Local transcription is the piece that's working. Everything past the transcript is still being built.
If you've dealt with local Whisper deployment on shared GPU, curious what you found: do you bother with beam_size tuning, or does the accuracy delta not matter enough at inference time to be worth it?
