How to Run Qwen 35B on 16GB VRAM [2026]: Flags + Quants

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

If you want how to run Qwen 35B on 16GB VRAM in 2026, here’s the outcome: you can get a “35B-class” Qwen model running on a consumer 16GB card without CPU-offload. The prerequisite that trips people up is KV cache. If you don’t budget it, your model loads, you generate 50 tokens, and then you hit an OOM spike and blame the runtime.

This post is a recipe. You’ll get:

• which “Qwen 35B” people actually mean in 2026
• quantization picks that fit 16GB with room for runtime overhead
• KV-cache math + a context-length table you can use before you download anything
• working flag sets for llama.cpp and vLLM that avoid the usual fragmentation/OOM traps
• a tiny benchmark harness that logs tokens/sec and peak VRAM

I’m opinionated about this: 16GB is enough for a 30B–40B dense model if you treat VRAM like a budget, not a vibe.

What is “Qwen 35B” in 2026?

“Qwen 35B” is a search shorthand for running a Qwen-family large language model in the ~30B–40B parameter class (the model size people want for “close to flagship” quality) on consumer GPUs.

In practice, most “Qwen 35B” queries map to Qwen2.5-32B-Instruct (32B parameters), because it’s a common sweet spot: big enough to feel meaningfully better than 7B/14B, small enough to squeeze onto 16GB with aggressive quantization.

If you’re picking a variant today:

• Prefer instruction-tuned for chat and coding prompts: Qwen2.5-32B-Instruct on Hugging Face.
• Prefer base only if you’re doing your own fine-tuning or controlled evals.

Here’s the canonical reference for the instruct model: Qwen/Qwen2.5-32B-Instruct.

Two adjacent realities worth saying out loud:

1. The “35B” number is not magic. Your quality comes from the specific model family, training recipe, and your quantization, not the integer in the name.
2. On 16GB, context length is a first-class tradeoff, not a checkbox. 16k context is possible, but it changes everything.

If you want broader model-selection context, I keep a running set of picks in local LLM guides and the more opinionated list in best local model for agentic coding.

Which quantizations fit in 16GB (and which ones are a trap)

Most guides say “use 4-bit” and leave it there. That’s how you end up with a model that technically runs but is unstable at your target context.

Weight memory: the boring math

A 32B model has ~32 billion parameters.

• FP16 weights: 32B * 2 bytes ≈ 64 GB (not happening on a 16GB card)
• 8-bit weights: 32B * 1 byte ≈ 32 GB
• 4-bit weights: 32B * 0.5 bytes ≈ 16 GB

That last line is why people fixate on 4-bit. But 16GB for weights alone leaves you zero headroom for KV cache, CUDA workspace, kernels, and allocator fragmentation. In the real world, you want weights to land closer to 12–14 GB if you want the rest of the system to breathe.

That’s why my default recommendations for 16GB are:

• GGUF (llama.cpp): Q4_K_M for “it fits and doesn’t feel awful”
• GGUF (llama.cpp): Q5_K_M only if you accept smaller context (or partial GPU offload)
• GPTQ 4-bit / AWQ 4-bit (vLLM/exllama-ish stacks): good for throughput, but pick conservative max length

If you want a deeper primer on formats, I already wrote GGUF vs GPTQ vs EXL2: LLM quantization compared and LLM quantization levels compared: Q4_K_M vs Q8_0 vs FP16.

Quality expectations (no mythology)

4-bit quantization quality loss is real, but it’s not uniform.

• For many model families, Q4 is “surprisingly fine” on instruction-following.
• For others, Q4 turns into brittle reasoning and weird refusals.

Based on the benchmark data I maintain at kunalganglani.com/llm-benchmarks, the pattern I keep seeing across model families is that a blanket “Q4 is always fine” recommendation is wrong. There are quality cliffs, and they’re model-specific. (This is exactly why I’m obsessed with reproducible harnesses, which you’ll get later in this post.)

If you’re optimizing for coding help and you can tolerate slightly lower context, I’d rather run a better quant (Q5-ish) at 4k–8k than a worse quant at 16k. Tokens you can’t use effectively are wasted tokens.

How to compute VRAM usage (weights + KV cache + runtime)

If you only take one thing from this post, take this: your OOMs are usually KV cache, not weights.

What is KV cache?

KV cache (key-value cache) is the memory the model uses to store attention “history” for each generated token, so it doesn’t have to recompute attention over the full prompt every time.

KV cache grows roughly linearly with:

• context length (prompt + generated tokens)
• number of layers
• hidden size
• number of KV heads (or grouped-query attention setup)
• bytes per element (FP16 vs FP8-ish vs quantized cache, depending on runtime)

The safe budgeting model I use

On a 16GB GPU, I budget like this:

• 12–14 GB for weights (depending on quant)
• 1–3 GB for KV cache (depends on context)
• 1–2 GB for runtime overhead (CUDA graphs, workspaces, fragmentation, driver overhead)

That last line is why “4-bit equals 16GB” is a trap. You need slack.

Practical VRAM budget table (16GB target)

These are conservative “don’t pagefault, don’t explode” targets. The exact KV cache size depends on the runtime and cache dtype, but the trend is stable.

When people say “it loads but OOMs when generating”, they’re typically trying 8k–16k on a setup that had no slack.

If you’re doing long-context Retrieval-Augmented Generation (RAG), read RAG context window limits. Bigger context is often the wrong lever anyway.

Pick a serving stack for 16GB (and the flags that actually prevent OOM)

I’ll be blunt: if your goal is “get Qwen 35B-class running on 16GB with minimum suffering,” start with llama.cpp GGUF. Then graduate to vLLM if you need multi-user serving and higher throughput.

I’ve shipped enough systems around deterministic gates in my own pipeline (the one that publishes this blog) to trust one principle: simple, observable systems beat clever systems when you’re chasing tail latencies and weird failure modes. The same applies to local LLM serving.

Option A: llama.cpp (GGUF) for single-user or small-team

llama.cpp is the most reliable “it just runs” stack for constrained VRAM.

Official repo: llama.cpp by Georgi Gerganov.

Flags that matter on 16GB:

• --ctx-size / -c: your context window. This is the KV cache lever.
• --n-gpu-layers: how many layers to offload to GPU. Use this to fit when you’re barely over budget.
• --batch-size (or prompt processing batch): too high increases peak memory.
• --flash-attn (if available in your build): can improve speed, but can change memory behavior.

A starting command pattern (edit paths, and start at 4096 context):

./llama-server \
  -m /models/qwen2.5-32b-instruct-q4_k_m.gguf \
  -c 4096 \
  --n-gpu-layers 999 \
  --threads 8 \
  --port 8080

If you OOM, don’t randomly change 10 flags. Do this sequence:

1. Drop -c from 8192 → 4096 (largest impact, most predictable)
2. If you still OOM, reduce GPU offload: --n-gpu-layers 999 → 60 → 40
3. If you still OOM, choose a smaller quant (Q4 over Q5)

I care about reproducibility here because I’ve seen “fixes” that only work until a different prompt shape hits production. Your goal is not “it ran once.”

If you want a GUI wrapper, see my take on Ollama vs LM Studio 2026 and Ollama vs llama.cpp 2026.

Option B: vLLM (AWQ/GPTQ) for throughput and multi-user serving

If you want to serve multiple users, you want vLLM. It’s a real serving engine with scheduling, batching, and memory efficiency.

Official docs: vLLM and the GitHub repo vLLM by vLLM contributors.

The flag that matters most on 16GB is the one that stops vLLM from greedily allocating all memory:

• --gpu-memory-utilization (set a cap, like 0.85 or 0.90)

A pragmatic server command pattern:

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-32B-Instruct \
  --dtype half \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.88 \
  --port 8000

Concrete numbers that matter:

• 0.88 means “use at most 88% of VRAM”, leaving ~12% for driver/runtime overhead.
• --max-model-len 4096 is not conservative for fun. It’s because 16GB + long context is where you’ll see allocator fragmentation and sudden spikes.

If you need a broader serving decision framework, I wrote how to serve a local LLM to multiple users.

Option C: ExLlamaV2 / EXL2 (when you want max speed per VRAM)

EXL2-style quantization can be extremely fast for single-GPU inference. The tradeoff is operational complexity and “it depends” behavior across kernels and driver setups.

If you’re already in the exllama ecosystem, your main lever is still the same: context length drives cache. Cap it early, then increase.

I’m not linking a specific command here because exllama packaging shifts faster than most blog posts get updated, and “copy-pasteable but wrong” is worse than no snippet.

If you want the format comparison, see GGUF vs GPTQ vs EXL2.

What breaks on consumer GPUs (and how to debug OOMs without guessing)

The annoying part about 16GB is not “it’s small.” It’s that you’re operating near the cliff, so tiny overheads matter.

Here are the failures I see most often:

1. Windows WDDM overhead: you have 16GB on paper but less in practice. A 0.5–1.5 GB hit is normal depending on what else is running.
2. Allocator fragmentation: you have enough free VRAM total, but not enough contiguous space for a big allocation.
3. CUDA graph spikes: graphs can increase memory usage for speed. Great when you have headroom. Pain when you don’t.
4. Flash-attn / attention kernel mismatch: kernels that improve throughput can change workspace needs.
5. Driver/toolkit mismatch: you “fixed” it by reinstalling, but you didn’t learn what broke.

My debugging loop is boring on purpose:

• Log peak VRAM for every run.
• Change exactly one variable at a time: quant, context, or offload.
• Keep a “known good” baseline prompt to compare.

If your workload is agentic and tool-heavy, you’ll also want to think about LLM cost and request shaping even when you run locally. Local doesn’t mean free. It means you pay in watts and latency. See local LLM break-even math and LLM cost.

A tiny benchmark harness: tokens/sec + peak VRAM (markdown output)

Most benchmarks are not benchmarks. They’re screenshots.

Here’s a minimal harness approach that works across stacks:

• fixed prompt
• fixed max generated tokens (e.g. 256)
• capture tokens/sec from the runtime output
• capture peak VRAM from nvidia-smi
• emit a markdown table so you can paste results into notes or issues

1) Fixed prompt file

Create prompt.txt with a stable prompt. Keep it consistent across runs.

Example prompt (short enough to fit in 4k even with system tokens):

• ask for a design doc
• ask for edge cases
• ask for a small code sample

2) Run + log peak VRAM

A shell harness pattern:

#!/usr/bin/env bash
set -euo pipefail

MODEL="$1"      # path or HF id
CTX="$2"        # e.g. 4096
OUT="$3"        # output label

# Sample peak VRAM every 200ms during the run
( while true; do nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits; sleep 0.2; done ) \
  | awk '{ if ($1>max) max=$1 } END { print max }' \
  > "${OUT}.peak_vram_mb" &
SMI_PID=$!

START=$(date +%s)

# Example: llama.cpp CLI (adjust to your binary)
./llama-cli -m "$MODEL" -c "$CTX" -n 256 -f prompt.txt \
  2>&1 | tee "${OUT}.log"

END=$(date +%s)
kill $SMI_PID || true

echo "seconds=$((END-START))" > "${OUT}.time"

3) Produce a markdown table row

Parse out:

• tokens/sec from the runtime log
• peak_vram_mb from the sampled file

Then you can build a table like:

The point is not perfect scientific rigor. The point is to stop guessing.

If you want to do benchmarking “for real”, read local LLM benchmark methodology and LLM latency benchmark methodology.

My default 16GB recipe (the one I’d tell a teammate to use)

If you’re on an RTX 4080 16GB or any similar 16GB GPU, and your goal is “Qwen 35B-class, stable, useful”:

1. Use Qwen2.5-32B-Instruct.
2. Start with GGUF Q4_K_M in llama.cpp.
3. Set --ctx-size 4096. Earn 8192 with real logs.
4. If you need serving, move to vLLM and cap --gpu-memory-utilization to 0.85–0.90.
5. Benchmark every change. Don’t trust vibes.

One more thing. The local AI crowd keeps trying to brute-force context length as if it’s free. It’s not. My prediction: in 2026, the winners won’t be the people running 32B at 16k context on a 16GB GPU. It’ll be the people who run 32B at 4k–8k with tight retrieval, good prompts, and a benchmark harness that stops regressions from shipping.

Originally published on kunalganglani.com