The first time I tried loading Whisper, bge-m3, and gemma at the same time on my local box, it OOMâd immediately. Iâd known this was going to happen, but I tried anyway to see where the ceiling actually was.
The machine is a workstation I already had, enough VRAM for any single model but not three simultaneously. I had some options: get more hardware, split across multiple machines, or figure out a sequential loading pattern. I went with sequential because adding hardware or managing separate boxes felt like problems I didn't want to solve yet.
Sequential loading means: load whatever model you need, use it, unload before moving on. Nothing runs in parallel. This was fine for my use case because the workloads don't actually overlap in time â I'vm not running embedding lookups while transcribing a recording. The load time overhead adds a few seconds per task. For Whisper, that's ~4 seconds on top of ~90 seconds of transcription for a 30-minute call, which I don't notice in practice.
The bge-m3 decision took the most deliberation
I started with all-MiniLM-L6-v2 for embeddings. It's small and fast. The issue was Korean/English mixed documents. I'm processing meeting summaries and reference docs that switch between the two, and MiniLM's cross-lingual similarity scores were unreliable enough that I was getting wrong nearest-neighbor results. bge-m3 handles cross-lingual matching better. It costs more VRAM and I had to drop the batch size from the default to stabilize it, but the accuracy difference on my actual data was clear.
One thing I didn't fully account for: when bge-m3 and Whisper are both unloaded, the load time for whichever comes next varies. bge-m3 seems to be slower on first load than subsequent loads in the same session, probably something to do with model weights caching at the OS level. I haven't investigated this properly.
Whisper was the easy pick
medium.en. I ran large once to compare on a client call recording and the accuracy improvement wasn't worth the extra VRAM cost for my use case. Transcription time for a 30-minute recording is under two minutes with medium. That's fast enough that I'm not sitting watching it.
Gemma was a coin flip between a few candidates
I needed local image analysis for screenshots and scanned documents. I tested gemma, a couple of LLaVA variants, and MiniCPM-V on maybe 20 images from my actual use case. Counted how many each model described correctly. Gemma came out ahead on document-heavy images. That test was informal enough that a different set of images might have given different results, but I had to pick something.
One thing I noticed later: gemma's processing time varies a lot by input image resolution. Standard screenshots are fast. A high-res photo from a phone camera takes noticeably longer. I still haven't profiled exactly what's happening there. If you're planning around latency, test with the actual image sizes you'll be using, not benchmarks from smaller inputs.
Where things stand
The sequential loading setup is stable. Iâve been running it for a few weeks without the OOM errors from the first attempt. The models work individually and I have code that chains them in sequence for a given input set.
What Iâm still building is the layer that makes use of this: a pipeline that processes call recordings, emails, and messages together and produces something more useful than three separate outputs. The individual pieces are working. Connecting them into a coherent pipeline is the current work.
Originally posted at hannune.ai
