Qwen3.8 27B VRAM: how to fit 262K context in 16 GiB, not 64

Qwen3.8 27B VRAM planning goes wrong at the first step: people multiply 64 layers by a per-layer KV cache and budget for 64 GiB at the model's full 262,144-token context. Only 16 of those layers cache anything — the other 48 are linear-attention layers carrying a fixed-size state — so one full-context sequence actually costs 16 GiB. That single architectural fact is the difference between "needs a multi-GPU node" and "runs on the card you already have."

TL;DR

• The FP8 checkpoint is ~25.9 GiB of weights (27.78B params at one byte each), plus KV cache.
• KV cache at 262,144 tokens: 16 GiB at BF16, 8 GiB at FP8 — because only 16 layers cache.
• A same-shaped model with all 64 layers doing full attention would need 64 GiB at the same context.
• The 48 Gated DeltaNet layers carry a constant ~72 MiB of state, regardless of context length.
• On a 48 GB card, --kv-cache-dtype fp8 is not optional — it is what makes 262K fit at all.

What is Qwen3.8 27B's hybrid attention stack?

Qwen3.8 27B is a dense 27.78B-parameter model whose 64 layers repeat a 3:1 pattern — three Gated DeltaNet linear-attention layers for every one grouped-query full-attention layer. The model card writes the layout as 16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN)), which works out to 48 linear layers and 16 full-attention layers, and the config confirms it with full_attention_interval: 4.

That ratio is not a benchmark-chasing detail. It is the memory model. Full attention has to remember every token it has seen, so its cache grows linearly with sequence length. Linear attention compresses history into a fixed-size recurrent state, so it remembers in constant space. Mixing them means three quarters of the depth stops participating in the part of the cost that scales.

The KV cache math you can actually check

Every number below comes from the published config.json. The relevant fields:

The KV cache formula for a grouped-query attention layer is unglamorous:

bytes/token/layer = 2 (K and V) × num_kv_heads × head_dim × bytes_per_element
                  = 2 × 4 × 256 × 2   (BF16)
                  = 4,096 bytes = 4 KiB

Multiply by the 16 layers that actually cache and you get 64 KiB per token. At the full window:

The right-hand column is the counterfactual worth internalizing. A dense 64-layer model with an identical GQA shape — 4 KV heads, 256 head dim — would burn 64 GiB to hold one full-context sequence. Qwen3.8 pays 16 GiB for the same window because 48 of its layers opted out of caching entirely.

What the linear layers cost instead

They are not free, they are just flat. Each Gated DeltaNet layer keeps a recurrent state shaped by its head dimensions — linear_num_value_heads: 48, linear_key_head_dim: 128, linear_value_head_dim: 128. That is 48 × 128 × 128 = 786,432 elements, about 1.5 MiB per layer at BF16, or roughly 72 MiB across all 48 layers (plus a negligible convolution state at linear_conv_kernel_dim: 4).

Seventy-two megabytes. At 262K tokens. It does not move if you go to a million.

Put differently: if those 48 layers had been full-attention layers, they would have cost 48 GiB at the same context. They cost 72 MiB. That is the trade the architecture is making, and it is why "how many layers does this model have" tells you almost nothing about its serving footprint anymore.

The Qwen3.8 27B VRAM budget, per card

Weights first. 27.78B parameters at one byte each in FP8 is 27.8 GB ≈ 25.9 GiB, with a small amount on top because the vision tower and the LM head are kept at higher precision. Then add the cache. Assume ~3 GiB for activations, CUDA graphs, and the runtime.

The last column is where this stops being trivia. On an 80 GB card, roughly 48 GiB is left for cache after weights and overhead. At 16 GiB per full-context sequence you get two. At 8 GiB you get five. Halving the cache dtype did not make the model faster — it made your batch 2.5× wider, which is the thing that actually moves tokens per second per dollar. The same principle drives most of the wins in tuning vLLM's throughput flags.

Serving it: the flags that matter

vLLM 0.17.0+ supports the model, and its published recipe is worth reading before you invent your own launch line:

vllm serve Qwen/Qwen3.8-27B-FP8 \
  --tensor-parallel-size 4 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3

Three of those four flags are memory decisions, not performance decisions:

1. --kv-cache-dtype fp8 is in every configuration vLLM publishes for this model, including the single-GPU low-latency one. That is a strong hint: at 262K, the maintainers do not consider a BF16 cache the default.
2. --tensor-parallel-size 4 on a 27B model looks like overkill until you read it as a memory move. Sharding 25.9 GiB of weights four ways frees ~19 GiB per device for cache. The recipe's own note calls this the configuration for the largest KV cache.
3. --max-model-len 262144 is a hard allocation ceiling, not a suggestion. vLLM reserves the cache up front. If you serve 32K workloads, setting this to 262144 hands 87% of your cache budget to a context you never use.

For speculative decoding the model ships an MTP draft head, enabled with --speculative-config '{"method":"mtp","num_speculative_tokens":3}'. And the 1M-token extension needs an explicit override, because the checkpoint's own max_position_embeddings is 262,144:

vllm serve Qwen/Qwen3.8-27B \
  --max-model-len 1010000 \
  --hf-overrides '{"text_config": {"max_position_embeddings": 1010000}}'

Run the numbers before you run that command. At 1M tokens the FP8 cache alone is 30.8 GiB, so a single sequence needs ~57 GiB and an 80 GB card serves exactly one of them.

Four mistakes this architecture invites

1. Sizing from parameter count. "27B at FP8 is 28 GB, so a 48 GB card is fine" is the reasoning that produces an out-of-memory error at the first long request. Weights are the floor; the cache is the part that varies by 20× depending on your --max-model-len.

2. Sizing from layer count. 64 layers is the number people multiply by. Sixteen is the number that matters. Any hybrid model — and there are more of them every quarter — breaks the old heuristic that depth and cache footprint move together.

3. Setting --max-model-len to the maximum by reflex. The cache is pre-allocated. Declaring 262K when your p99 request is 24K does not give you flexibility, it gives you a smaller batch. Set it to what you actually serve, plus margin.

4. Treating --kv-cache-dtype fp8 as free. It is a genuine quantization of the attention history. Chat and code completion usually shrug it off; long-context retrieval is where it shows up first — which is exactly the workload you bought a 262K window for. Test it at your target length rather than assuming, the same way you would validate any context-management strategy that decides what survives.

Where this leaves the 27B class

The interesting comparison is not against bigger models, it is against same-sized ones. Running a 30B model like Muse Glimmer locally is a weights-dominated problem: you fight for room to hold the parameters, and context is a rounding error. Qwen3.8 27B inverts that at long context. Its weights are ordinary; its cache is what you engineer around, and the architecture has already done three quarters of that work for you.

That is also why offload tricks like running a 70B model on a 4 GB GPU don't transfer here. Those techniques stream weights. The bottleneck in a 262K-context hybrid model is a cache that has to stay resident, and there is nothing to stream. If you're memory-bound on a laptop-class device the fix is a smaller window, not a cleverer loader — the same conclusion that shows up when diagnosing slow LLM inference inside macOS VMs.

The one number to remember

64 KiB per token. That is Qwen3.8 27B's marginal KV cost at BF16, and 32 KiB at FP8. Multiply by your real p99 context length, multiply by your target concurrency, add 25.9 GiB, and you have your card. Everything else in this post is derivation.

If you're standing up a serving stack around this, the flag-level companion to this post is vLLM throughput tuning — this one tells you what fits, that one tells you how fast it goes.

Frequently asked questions

How much VRAM does Qwen3.8 27B need?

The FP8 checkpoint is about 25.9 GiB of weights, and a single sequence at the full 262,144-token context adds 16 GiB of KV cache at BF16 or 8 GiB with --kv-cache-dtype fp8. That puts a realistic single-sequence floor at roughly 34 GiB with an FP8 cache and roughly 42 GiB with a BF16 one, before activations. An 80 GB card runs it comfortably; a 48 GB card only works if you quantize the KV cache.

Why is Qwen3.8 27B's KV cache smaller than other 27B models?

Only 16 of its 64 layers use full attention. The other 48 are Gated DeltaNet linear-attention layers, which carry a fixed-size recurrent state instead of a per-token key/value history. A conventional model with the same 64 layers and the same GQA shape would need 64 GiB of KV cache at 262K tokens — four times as much — because every layer would be caching.

What does --kv-cache-dtype fp8 actually cost in quality?

It halves the per-token cache footprint by storing keys and values in 8-bit instead of 16-bit. The degradation is usually small for chat and coding workloads, but it is not free: long-context retrieval and needle-in-a-haystack style tasks are the first places it shows up. Benchmark your own evaluation set at your target context length before assuming it is a no-op.

Does the Gated DeltaNet state grow with context length?

No, and that is the entire point of the architecture. Each linear layer keeps a fixed recurrent state sized by its head dimensions — about 1.5 MiB per layer at BF16, roughly 72 MiB across all 48 layers — whether you feed it 8,000 tokens or 1,000,000. Memory growth with sequence length comes only from the 16 full-attention layers.

Can I run Qwen3.8 27B at the full 1M-token context on one GPU?

Not with a BF16 cache. Extending to 1,010,000 tokens with YaRN pushes the KV cache to about 61.6 GiB at BF16 or 30.8 GiB at FP8, on top of 25.9 GiB of weights. The FP8 path lands near 57 GiB, which fits an 80 GB card for a single sequence and nothing else. Multi-sequence serving at 1M context needs tensor parallelism across several GPUs.

Is tensor parallelism worth it for a 27B model on one node?

For throughput at long context, yes. vLLM's own recipe uses --tensor-parallel-size 4 for the FP8 checkpoint specifically to free VRAM for the cache: splitting 25.9 GiB of weights across four GPUs leaves far more room per device for concurrent sequences. At short contexts, TP1 is the better latency choice because you avoid the collective communication overhead.

Sources

• Qwen/Qwen3.8-27B-FP8 model card — architecture, layer layout, context length, license.
• Qwen3.8-27B config.json — every dimension used in the KV-cache arithmetic above.
• vLLM recipe for Qwen3.8-27B — the published serving commands and flags.

Originally published at umesh-malik.com

Keep reading on umesh-malik.com:

• vLLM throughput tuning: configure these four flags, not a bigger GPU
• Fix slow LLM inference in macOS VMs: 12.6 → 207 tok/s
• Run 70B LLM on 4GB GPU: AirLLM's Real Tradeoff