Qwen3.5-9B spends 32KB per token on KV cache

Alibaba's Qwen3.5-9B uses full attention on only 8 of its 32 layers. The other 24 use a linear method whose memory cost does not grow with the conversation. That split is why a 9-billion-parameter model can hold a long chat on an 8GB graphics card.

The model shipped in February 2026 under the Apache 2.0 license. Most coverage focused on its test scores. VentureBeat reported that it scored 81.7 on GPQA Diamond, a graduate-level science reasoning test, ahead of OpenAI's gpt-oss-120b at 80.1. For anyone running the model on their own hardware, the layer layout matters more than the score.

What a KV cache is and why it grows

A language model re-reads the whole conversation before it writes each new word. Redoing that work every time would be slow. So the model saves intermediate values, called keys and values, in a store known as the KV cache.

The weights of a model are a fixed cost. The KV cache is not. It grows with every token in the conversation.

On a machine with 8GB of memory, that difference decides a lot. A model file may load fine and still run out of room once a chat gets long. When that happens, the software usually moves layers to the main processor instead of failing outright, and speed drops sharply.

How Qwen3.5-9B splits its layers

The Qwen3.5-9B model card describes the stack as 8 repeats of a 4-layer block. Three layers in each block use Gated DeltaNet, a linear attention method. The fourth uses gated full attention.

Linear attention keeps a fixed-size running state. It does not grow as the text gets longer. So those 24 layers cost the same at 4,000 tokens as they do at 200,000.

Only the 8 full-attention layers hold a cache that grows. The model card gives their shape: 4 key-value heads, each 256 dimensions wide.

That is enough to work out the cost per token. Multiply the 8 layers by 2, because keys and values are both stored, then by 4 heads, then by 256 dimensions, then by 2 bytes per value. The result is 32,768 bytes, or 32KB for every token in the conversation.

What that costs at real context lengths

The model card lists a native context length of 262,144 tokens, extensible to about 1,010,000. At 32KB per token, the cache cost at each length is straightforward arithmetic.

The second column below shows what the same model would cost if all 32 layers used full attention instead of 8. It is a hypothetical, included to show the size of the saving.

Context length	Qwen3.5-9B KV cache	If all 32 layers used full attention
8,192 tokens	0.25 GiB	1.0 GiB
32,768 tokens	1.0 GiB	4.0 GiB
131,072 tokens	4.0 GiB	16.0 GiB
262,144 tokens	8.0 GiB	32.0 GiB

The design saves four times the cache memory at every length. On an 8GB card holding roughly 6.6GB of 4-bit weights, that is the difference between a 32,000-token conversation and a 4,000-token one.

Qwen has since pushed the same idea further. Its Qwen3.8-Flash-Next release uses Gated DeltaNet in three of every four layers as well, alongside a sparse expert design.

What this means for developers

Parameter count is a poor guide to whether a model fits in a given amount of memory. Two models of the same size can differ by four times or more on cache cost alone. Check the layer layout before trusting a size estimate.

The numbers to look for are in the model's published configuration: how many layers use full attention, how many key-value heads each one has, and how wide each head is. Those three numbers, times two for keys and values and two for 16-bit storage, give the cost per token. Multiply by the context length you actually plan to use.

Budget for the cache before choosing a context window. A common mistake is setting the context to the model's maximum because the maximum is available. Most local runtimes accept a setting that does not fit and then quietly move work to the processor, which looks like a mysterious slowdown rather than an error.

Halving the cache is also an option. Most runtimes can store the cache at 8-bit precision instead of 16-bit, which cuts the figures in the table above in half at a small cost in quality. On an 8GB machine that is often the setting that turns a model from unusable to comfortable.

This article was first published on Tech AI Wire.