Run Qwen 3.8 27B Locally (2026): 17 GB Total, Not 17 GB VRAM

The number circulating for Qwen3.8-27B is 17 GB, and it comes straight from Unsloth's own hardware table. Read the table header and it says something more specific than the number implies: units are total memory, RAM plus VRAM, or unified memory. That distinction is the difference between a model that runs on your card and a model that runs partly on your motherboard.

What You Can Run Locally (and What You Can't)

With 24 GB you run the quant Qwen intended; with 16 GB you run a compromise; below that, do not bother. Qwen released Qwen3.8-27B under Apache 2.0 on Hugging Face, a dense 27B vision-language model with a 262,144-token native context. It is small enough that the question stopped being "which datacenter" and became "does this fit the machine on my desk." That is a different conversation from the one around GLM 5.2, where 2-bit still wants a 256 GB Mac Studio. Here is the short version.

Weights license:        Apache 2.0, commercial use allowed
Parameters:             27B dense (llama.cpp counts 27.32B)
Native context:         262,144 tokens (1M via YaRN)
Smallest usable GGUF:   9.01 GB (Unsloth UD-IQ2_XXS)
4-bit GGUF:             16.81-18.97 GB depending on publisher
Vendor memory guidance: 11-13 GB at 2-bit, 17-19 GB at 4-bit, total RAM+VRAM
Vision:                 separate mmproj file, +0.63-0.93 GB by publisher
Measured on M2 Pro 16GB: 7.11 tok/s generation at 2-bit, 74.59 tok/s prompt
Same box at 4-bit:      prompt runs 4x slower, generation fails outright
Setup time:             about 20 minutes, most of it downloading

What you can do after this setup: run a strong coding and agentic model offline, with no per-token bill and no data leaving the machine. What you cannot do: match hosted latency, run the full 256K context on a consumer box, or get the Max tier, which has no open weights at all.

The honest headline: a 24 GB card is the entry point for the quant Qwen actually wants you to use, and 32 GB is where it stops being a squeeze. Below 16 GB the interesting question is not which quant, it is whether local inference is the right call at all.

How Much VRAM Does Qwen 3.8 27B Actually Need?

Between 11 GB and 19 GB of total memory, and "total" is doing the work in that sentence. Unsloth's hardware table labels its units explicitly: total memory, meaning RAM plus VRAM, or unified memory on Apple silicon. It is a budget for the whole machine, not a spec for the graphics card.

The guidance sits a couple of gigabytes above each file because the file is not the only thing in memory. You also pay for the KV cache, the compute buffers, and whatever your OS is already holding.

Where this gets misread: the same Unsloth page says 4-bit "will work on most devices 17-19GB VRAM like RTX 5080, 4090 or a Mac with 24GB RAM." Per NVIDIA's own comparison page, the RTX 5080 ships with 16 GB of GDDR7, as does the 5070 Ti. The 4090 has 24 GB and the 5090 has 32 GB. So on a 5080 the 4-bit path is a total-memory budget being met by the card plus system RAM, exactly as the table header says, and not 17 GB sitting in VRAM. It still runs. Some of the layers just live on the wrong side of the PCIe bus, and generation speed follows them there.

If you want a single rule, use Unsloth's: RAM plus VRAM should be at least the size of the quant file, otherwise it works but pages off disk and slows down badly.

Which GGUF Quant Should You Download?

Take the largest file that leaves you 2-3 GB of headroom, and compare byte counts rather than quant names. The name is not a size. Three publishers shipped a file called Q4_K_M for this model and they are 2.2 GB apart, because each one chooses different per-tensor precision under the same label.

On a 16 GB card that spread is the whole decision. The LM Studio build is 300 MB under Unsloth's and 2.2 GB under ggml-org's, and none of that is visible from the quant name.

Practical picks by memory budget:

• 32 GB and up: UD-Q4_K_XL at 17.92 GB, or Q6_K at 22.88 GB if you want to spend the headroom on precision instead of context.
• 24 GB: Q4_K_M from lmstudio-community at 16.81 GB leaves the most room for KV cache.
• 16 GB: UD-Q3_K_XL at 13.44 GB fits with room to work. The 4-bit files do not, without offload.
• 16 GB unified Mac: UD-IQ2_XXS at 9.01 GB. This is the floor, and you will feel it.

Unsloth's UD prefix marks their dynamic quants, which keep sensitive layers at higher precision instead of quantizing uniformly. At the 2- and 3-bit end that matters more than the nominal bit count.

When Is Running Qwen 3.8 27B Locally Worth It (and When Is It Not)?

When the data cannot leave the building, or when your usage is heavy enough that a GPU amortizes. Otherwise the hosted path is cheaper and considerably faster.

Use the local setup when:

1. You are working with code or documents under a contract that forbids third-party inference.
2. You already own a 24 GB or 32 GB GPU and want to stop paying per token for routine refactors and reviews.
3. You need it offline, on a plane, in an air-gapped lab, or behind a network you do not control.

Do not use the local setup when:

1. Your card has 12 GB or less. The quants that fit are not good enough to be worth the setup.
2. You want the top of the family. Qwen3.8-Max is API-only with no open weights, and the 2.4T-A95B sibling needs 397 GB at 1-bit.
3. You are doing long agentic runs. At the 2.91 tok/s measured below on a real prompt, a task a hosted model finishes in four minutes runs for most of an hour.

Stop rule: if you get through step 4 below and generation is under 5 tok/s at the context length you actually use, stop tuning. That machine is not the right host for this model, and no flag will change it by an order of magnitude.

What Hardware Runs Qwen 3.8 27B?

A 24 GB GPU or a 32 GB Mac is the comfortable entry point; 16 GB works with compromises. The card matters less than the total memory pool and its bandwidth.

Software side, llama.cpp is less of a hurdle than people expect. The architecture string in config.json is qwen3_5, and llama.cpp has supported that family since February 2026, when PR #19435 landed dense and MoE support. Testing that here: build b10375, published 2026-08-12 at 12:18 UTC, before the weights themselves landed on Hugging Face at 08:23 UTC the following day, loads and runs the file without complaint. So "update llama.cpp" is rarely the fix, and if your build is recent enough to have run any Qwen 3.5 or 3.6 model, it will run this one. On macOS brew install llama.cpp is current enough; on Linux and Windows take a release binary or build from source.

One Apple silicon detail that is not in any vendor table. macOS does not hand the whole memory pool to the GPU. On the 16 GB M2 Pro used for this article, llama.cpp reported the Metal limit directly at startup:

ggml_metal_device_init: has unified memory    = true
ggml_metal_device_init: recommendedMaxWorkingSetSize  = 12713.12 MB

12.71 GB, not 16 GB. That single line decides which quants are candidates on any Mac, and it is worth reading before you download 17 GB of weights.

How Do You Run Qwen 3.8 27B in llama.cpp?

Five steps, and the download is the slow one. Everything below was run on the M2 Pro described above.

Step 1: Install llama.cpp

brew install llama.cpp
llama-cli --version

Expected result: a build string. Mine was b10450-ece963f41. As covered above, almost any 2026 build will do.

Step 2: Pick a quant and check it against your ceiling

curl -s "https://huggingface.co/api/models/unsloth/Qwen3.8-27B-GGUF/tree/main" \
  | python3 -c "import json,sys;[print(f\"{f['path']:32s}{f['size']/1e9:6.2f} GB\") for f in json.load(sys.stdin) if f['path'].endswith('.gguf')]"

Expected result: the full file list with byte-accurate sizes. Compare against your total memory, not your card.

Step 3: Download the weights

curl -L -o qwen38-27b.gguf \
  "https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/Qwen3.8-27B-UD-Q3_K_XL.gguf"

Expected result: one file, no shards at this size. Swap the filename for the quant you picked in step 2.

Step 4: Run it

llama-server -m qwen38-27b.gguf -c 16384 \
  --temp 1.0 --top-p 0.95 --top-k 20 --port 8080

Expected result: an OpenAI-compatible endpoint on localhost:8080. Those sampling values are Qwen's published settings for thinking mode. For non-thinking use --temp 0.7 --top-p 0.80 --top-k 20 --presence-penalty 1.5.

Step 5: Add vision, if you want it

The base GGUF is text-only. Load it alone and llama.cpp says so, printing modalities : text. The vision half lives in a separate projector file:

curl -L -o mmproj.gguf \
  "https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/mmproj-F16.gguf"
llama-server -m qwen38-27b.gguf --mmproj mmproj.gguf -c 16384 --port 8080

Expected result: image input accepted. Budget 0.93 GB for that F16 projector, on top of your weights, not inside the number you planned for. If you want the smaller 0.63 GB Q8_0 projector, ggml-org is the only one of the three publishers shipping it, so that file comes from a different repo than your weights.

How Fast Is Qwen 3.8 27B on a 16 GB Mac?

7.11 tokens per second at 2-bit, and the 4-bit file will not generate at all. Measured with llama-bench on the M2 Pro, llama.cpp b10450, both quants on the same machine:

The 4-bit row is the one worth sitting with. The file loads. The model reports its parameter count correctly. Prompt processing runs, four times slower than at 2-bit, because the weights no longer fit the 12.71 GB Metal working set and the machine is paging. Then generation dies with failed to decode generation batch, res = -3. llama.cpp's header documents any return below -1 as a fatal error, distinct from return code 1, which is the ordinary "could not find a KV slot" you get from an oversized batch. Going under the vendor's memory number does not degrade gracefully on this hardware. It gets you a benchmark that half completes.

Those are synthetic numbers at zero context. The real-world figure is worse. Serving the same 2-bit model through llama-server at -c 16384 and sending a 7,072-token prompt, llama.cpp's own timing report reads:

prompt eval time = 208988.11 ms /  7072 tokens ( 33.84 tokens per second)
       eval time =  27129.34 ms /    80 tokens (  2.91 tokens per second)
      total time = 236117.45 ms /  7152 tokens

Four minutes for one turn, with generation at 2.91 tok/s rather than the 7.11 the benchmark promised. Prompt processing degrades as the context fills. llama.cpp's running average, printed per progress step, reads 43.58 tok/s at token 2,090 and 37.82 by token 6,186. Any tokens-per-second figure quoted at zero context, including the ones in the table above, is the best case you will ever see.

That request also demonstrates the thinking default. It was capped at 80 completion tokens and came back with a fragment rather than an answer, because reasoning_effort defaults to xhigh and the reasoning consumed the budget before an answer existed. The signature is easy to check for: ask something that needs a few steps of reasoning, cap max_tokens at 80, and the response arrives with finish_reason: length, an empty content, and all 80 tokens sitting in reasoning_content. Ask it 84 * 3 instead and the same cap is fine, 29 to 51 tokens across three runs, because short reasoning fits. The failure is not the cap, it is the cap meeting a question the model wants to think about. On a machine this slow, set reasoning_effort to low or disable thinking for routine work, and give it a real max_tokens.

One quiet detail in the load log: llama.cpp prints model has unused tensor blk.64.nextn.* and skips it. That is the multi-token prediction head, which Qwen trained for faster inference and which this GGUF path does not use. ggml-org ships MTP weights as separate files. If you want that speedup, you have to go looking for it.

The quality cost at 2-bit shows up quickly. Asked for a linked-list merge, the model's thinking trace sketched the solution as malformed Python, def __init__(self, val): self.val; self.next, which is two no-op expressions rather than assignments. Thinking traces are drafts and the final answer may well have been fine, but this is the kind of slip that gets rarer as you move up the quant range. It is the strongest argument for finding 24 GB rather than making 16 GB work.

Two labels worth not being confused by. llama.cpp reports the architecture as qwen35, because Qwen3.8-27B is built on the Qwen3.5 architecture and config.json declares model_type: qwen3_5. It also reports Unsloth's dynamic 2-bit file as ftype Q4_K - Small, which is a header field, not the actual mix. Trust the byte count.

Why Does a 27B Model Need So Little KV Cache?

Because only 16 of its 64 layers use full attention. Qwen publishes the layer topology in the model card: 16 repetitions of three Gated DeltaNet blocks followed by one Gated Attention block. Gated DeltaNet is a linear-attention layer whose state is a fixed size per sequence, so it does not grow as the conversation does. Only the 16 full-attention layers keep a per-token cache.

Working from config.json, those layers use 4 KV heads at head dimension 256. At f16 that is 4 KiB per token per layer, and across 16 layers, 64 KiB per token:

A conventional 64-layer model with the same head configuration would need four times the cache, 64 GiB at full context. The hybrid layout is why a 27B model with a 256K window is a desktop proposition at all.

It also sets the honest answer on context. The weights support 262,144 tokens natively and stretch to a million with YaRN, but a full native context costs 16 GiB of cache by itself. On the 16 GB test machine, llama-server started and served requests at -c 16384 with the 2-bit quant, and 8K to 16K is the practical band there. On 32 GB you can hold 32K comfortably alongside a 4-bit model. Set -c to what you actually use, and if you need more, quantize the cache with --cache-type-k q8_0 --cache-type-v q8_0 for roughly half the footprint at a small quality cost. Reserve the full window for the hosted path, which is the same reasoning that applies to context windows generally.

Common Errors During Local Setup (and Fixes)

Most local failures here are memory failures wearing a different error message. Every row below was reproduced on the test machine except the last, which comes from Qwen's own best-practices note.

Can a Team Share One Local Qwen 3.8 27B Box?

One box serves one or two developers, not a team. llama-server exposes an OpenAI-compatible endpoint that any teammate can point a client at, and that part is genuinely easy:

llama-server -m qwen38-27b.gguf -c 16384 --host 0.0.0.0 --port 8080 --parallel 2

The limit is arithmetic. A single consumer GPU running a 27B model at 4-bit produces one stream of tokens, and --parallel divides your context budget among slots rather than multiplying throughput. Two developers on a 4090 will notice each other. Four will queue.

Teams that want a shared local endpoint should be looking at vLLM or SGLang on a datacenter card, which is a different project with a different budget. That path is laid out in the GLM 5.2 self-host vLLM hardware and cost guide; the sizing logic carries over even though the model does not. A 27B at 4-bit needs roughly a third of the memory a 48 GB card offers, which leaves real room for concurrent KV cache, so one such card is a more sensible purchase for a team than four 4090s that each hold a private copy of the weights.

What Do You Do When the Local Box Runs Out Mid-Task?

Point the client at a hosted endpoint that speaks the same protocol, and keep working. Every local setup has the same two gaps, and they are not llama.cpp's fault.

The first is capacity. Your box runs one model at whatever speed its memory bandwidth allows, and when a task needs the 2.4T sibling, the Max tier, or simply a faster answer, the local endpoint has nothing to offer. The second is coverage. Qwen3.8-27B has open weights; Qwen3.8-Max does not, and the model card points at a Qwen Cloud hosted 27B with a 1M default context that is described as coming soon, with the linked overview page still returning a 404 at the time of writing. Some of this family you cannot host at any hardware budget.

Because llama-server speaks the OpenAI Chat Completions shape and so do the hosted gateways, the fix is the same on either side: change base_url and the key, keep the code. A gateway like ofox.ai carries bailian/qwen3.8-max alongside the previous generation bailian/qwen3.6-27b and bailian/qwen3.5-27b, so the same client can fall back from your GPU to a hosted tier without a second account. The open-weights 27B specifically is not in that catalog at the time of writing; on OpenRouter two providers carry it, both at the full 262,144-token context: Chutes at $0.40 per million input tokens and $3.00 per million output serving fp8, and AkashML at $0.45 and $3.20 serving bf16. The same lesson as the GGUF table applies one layer up. The cheaper endpoint is the quantized one, and the price difference is the precision difference.

Run the numbers before buying hardware. At those rates a developer pushing 10 million input and 2 million output tokens a month pays $10 to $11 depending on which provider serves the request. A 4090 does not pay for itself against that; privacy and offline access are the reasons to own one, not arithmetic.

Is the 27B Worth Running at All?

Against its own predecessor, clearly yes. Qwen's published figures put Qwen3.8-27B at 73.0 on Terminal Bench 2.1 versus 63.4 for Qwen3.6 27B, 61.7 on SWE-bench Pro versus 53.5, and 79.0 on their in-house QwenSWEBench versus 49.3. On computer use, OSWorld-Verified moved from 63.9 to 84.3. These are vendor numbers run on the Claude Code harness with corrections applied to some task sets, so treat them as a direction rather than a leaderboard, but the direction is steep for one generation at the same parameter count. For an independent read on what the 27B size class does against a frontier hosted model, the Qwen 3.6 27B versus Claude Opus 4.6 coding comparison is the closest baseline we have.

The comparison that matters more for a local box is against the quant you can actually afford to run. A 4-bit 27B on a 24 GB card is close to the model Qwen benchmarked. A 2-bit 27B on a 16 GB laptop is not, and the gap between them is larger than the gap between generations.

There is an independent data point for what the extra memory buys. When the previous generation shipped, Simon Willison ran it locally and reported his own numbers on 2026-04-22: "I tried it out with the 16.8GB Unsloth Qwen3.6-27B-GGUF:Q4_K_M quantized version," logging generation at 25.57 tokens per second and calling it "an outstanding result for a 16.8GB local model." That is the same size class one generation back, at 4-bit, on a machine with room for it. The 16 GB Mac in this article gets 7.11 tok/s on a synthetic benchmark and 2.91 on a real prompt, at 2-bit. Between three and nine times slower, on a worse quant, for the sake of about 8 GB. Buy memory before you buy anything else.

References

• Qwen/Qwen3.8-27B model card
• Qwen3.8-27B config.json
• Unsloth: Qwen3.8 how to run locally
• unsloth/Qwen3.8-27B-GGUF file listing
• ggml-org/Qwen3.8-27B-GGUF file listing
• lmstudio-community/Qwen3.8-27B-GGUF file listing
• NVIDIA GeForce graphics card comparison
• OpenRouter: qwen/qwen3.8-27b
• llama.cpp
• Simon Willison: Qwen3.6-27B

Frequently Asked Questions

Can I run Qwen 3.8 27B on a 16 GB GPU?

Not at 4-bit, and not entirely on the GPU. The 4-bit GGUF is 16.8-19.0 GB depending on who packaged it, which is already larger than a 16 GB card before any KV cache. On a 16 GB card you either drop to a 3-bit quant (12.6-13.8 GB) or keep 4-bit and let llama.cpp offload the overflow layers to system RAM, which works but moves generation speed onto DDR bandwidth.

Is Qwen 3.8 27B multimodal when you run it locally?

Only if you download the projector file separately. The base GGUF ships text-only; llama.cpp reports 'modalities: text' when you load it alone. Vision needs the companion mmproj file passed with --mmproj, and it sits on top of your weights budget rather than inside it. Unsloth and lmstudio-community ship it at 0.93 GB; only ggml-org publishes a 0.63 GB Q8_0 projector, so you may need to pull it from a different repo than your weights.

What is the difference between the Unsloth, ggml-org and LM Studio Q4_K_M files?

Size, by up to 2.2 GB. The same Q4_K_M name is 16.81 GB from lmstudio-community, 17.11 GB from Unsloth and 18.97 GB from ggml-org, because each publisher picks different per-tensor precision under the same label. On a memory-constrained box that spread decides whether the model fits, so compare byte counts on the Hugging Face file listing rather than trusting the quant name.

Does Qwen 3.8 27B support the full 256K context on a local machine?

The weights do, your memory usually does not. The KV cache costs about 64 KiB per token at f16, so a full 262,144-token context needs roughly 16 GiB on top of the weights. That is fine on a 64 GB or larger box and impossible on a 16 GB one. Set -c to what you actually use, typically 8K to 32K, and quantize the cache if you need more.

How many tokens per second does Qwen 3.8 27B generate on Apple silicon?

On an M2 Pro with 16 GB unified memory running the 2-bit Unsloth quant under llama.cpp b10450, the synthetic benchmark gives 7.11 tok/s generation and 74.59 tok/s prompt processing at zero context. A real request with a 7,072-token prompt measured 2.91 tok/s generating and 33.84 tok/s on the prompt, four minutes for a single turn. Quote the second pair, not the first.

Is Qwen 3.8 27B open source?

Yes, the weights are on Hugging Face under Apache 2.0, which permits commercial use. That covers Qwen3.8-27B specifically. Qwen3.8-Max is API-only and Qwen Cloud lists a hosted 27B with a 1M default context as coming soon, so the open-weights path and the hosted path are not the same product.

Why does llama.cpp report the architecture as qwen35 for a Qwen 3.8 model?

Because Qwen3.8-27B is built on the Qwen3.5 architecture, and the config declares model_type qwen3_5. The version number in the product name moved; the layer topology did not. Nothing is wrong with your download.

Should I run the 27B locally or call Qwen 3.8 through an API?

Local wins on privacy, offline use and a fixed hardware cost. The API wins on speed and on the 2.4T and Max tiers you cannot host at all. Qwen3.8-27B on OpenRouter runs $0.40 to $0.45 per million input tokens and $3.00 to $3.20 per million output across its two providers, so a light user reaches break-even against a GPU purchase very slowly.

Originally published on ofox.ai/blog.