Every major LLM lab is in a conundrum today, deliberating between scale vs cost. Making a dense model bigger makes it smarter, yes, but also makes every token more expensive to generate. In a dense model, every parameter activates on every token, and the compute cost of a forward pass scales ~linearly with parameter count.
Mixture of Experts (MoE) challenges the status quo with its architecture. Instead of one giant feed-forward network per layer, an MoE layer holds many smaller so-called "expert" networks, and a learned gate activates only a handful of them per token per layer. The result is a model that can carry hundreds of billions of parameters in vRAM, while activating only a small fraction of them for each run. To give an idea of scale, the DeepSeek-V3 model holds 671 billion params, while MoE enables it to use only around 37 billion of them each time.
1. 1991: Where It Started - Adaptive Mixtures of Local Experts
The idea of "mixture of experts" predates deep learning as we know it. It was first proposed in a 1991 paper by Jacobs, Jordan, Nowlan, and Hinton, Adaptive Mixtures of Local Experts. The core idea: instead of training one large network to solve a hard task, train several smaller "expert" networks and a separate gating network that learns how to weight and combine their outputs, all trained jointly.
The paper's demonstration was a vowel discrimination task. Given the same speech data, a system of competing experts plus a gating network was compared against a single monolithic network. The learning procedure was shown to decompose the hard problem into sub-problems that each expert could handle individually.
There's an important architectural detail here that's easy to miss, and it matters for everything that follows: this original formulation is what we'd now call a dense-MoE. The gate produces a weight for every expert, and every expert computes an output for every input; they're just combined with different weights.
That single detail is why the 1991 paper sits as a conceptual ancestor rather than a direct blueprint for modern LLM MoE. It establishes gating as a learned, differentiable routing function, which is an idea that survives for the next three decades. However, it doesn't buy any compute savings, because all experts still run on every input.
Model Architecture of the original 1991 expert/gate paper
2. 2017: The Sparse-Gating Leap
The paper that actually bridges 1991 and modern LLMs is Shazeer et al.'s 2017 paper from Google Brain, Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.
The key change here: sparse gating. Instead of every expert computing on every input, the gate selects only the top-k highest-scoring experts (4/4096 in this scenario), and only those experts actually run. This is the shift that actually saves compute, and it's the mechanism every modern LLM MoE is based on.
A minimal version of the routing logic looks like this:
# Simplified top-k sparse gating
def moe_layer(x, experts, gate_weights, k=4):
logits = x @ gate_weights # [num_experts] raw scores
top_k_vals, top_k_idx = topk(logits, k)
top_k_weights = softmax(top_k_vals) # renormalized over just the top-k
output = 0
for weight, idx in zip(top_k_weights, top_k_idx):
output += weight * experts[idx](x) # only these experts run
return output
Need for a Load-Balancing Loss
Sparse gating introduces a new problem. If you train the gate network purely on the downstream task loss, it tends to collapse. Here's the failure mode: a small random edge in initialisation means one expert responds marginally better to some input than another. The gate notices, routes slightly more traffic there. That expert now gets more gradient updates than its neighbours, so it improves further, so the gate favours it even more.
Left unchecked, this rich-get-richer loop concentrates almost all traffic onto a handful of experts, while the rest stay undertrained and never develop anything useful.
Shazeer et al.'s fix is an auxiliary loss, a term added on top of the main task loss that penalises uneven expert usage across a batch, nudging the gate toward spreading tokens more evenly across the whole expert pool.
# Simplified load-balancing auxiliary loss
def load_balance_loss(gate_probs, expert_assignments, num_experts):
# fraction of tokens routed to each expert
frac_routed = count_per_expert(expert_assignments) / len(expert_assignments)
# average gate probability assigned to each expert
avg_gate_prob = mean(gate_probs, axis=0)
return num_experts * sum(frac_routed * avg_gate_prob)
3. 2020–2021: Google's MoE-for-Transformers Era
The 2017 paper proved sparse MoE worked in an LSTM. The next step was bringing it into the Transformer, inside the feed-forward (FFN) sub-layer of each transformer block, rather than replacing the whole layer.
GShard (2020) was the first major distributed-training MoE story at Transformer scale, using top-2 gating. Each token routed to its top 2 experts, and GShard combined this with techniques for sharding enormous expert pools across many devices.
Switch Transformer (2021) simplified this further, to top-1 gating, just one expert per token, per layer. The authors argued this was not just cheaper but more stable to train than top-2, while still scaling to trillion-parameter total capacity.
| GShard (2020) | Switch Transformer (2021) | |
|---|---|---|
| Gating | Top-2 | Top-1 |
| Main claim | Distributed sparse training at scale | Simpler routing = more stable + cheaper |
| Where it lives | Transformer FFN sub-layer | Transformer FFN sub-layer |
4. 2023: Mixtral and the Specialisation Question Gets Interesting
Mistral's Mixtral 8x7B (2023) is one of the first widely-used open-weight LLMs built on MoE, 8 experts per layer, top-2 gating, with routing decisions made independently at every layer for every token.
The Mixtral paper includes its own analysis of what the experts specialise in, and the finding is striking: at this scale, on general language modelling, they found no clean topical or domain specialisation. No expert cleanly owns "math," or "code," or "French." What structure they did find looked more syntactic or positional than semantic, plus some correlation between consecutive tokens routing to the same expert, which reads more like local context redundancy than deliberate task-based division of labor.
Proportion of Expert Distributions
What the Experts Actually Learned
The 2017 paper also did something that turns out to matter a lot for the rest of this story: it looked inside the trained model to see what each expert had specialised in. Table 9 of the paper, from a WMT'14 English→French translation model, shows real, identifiable clustering, but not along the lines you'd guess.
- One expert fires heavily on phrasing like "a leading/critical/central role"
- Another clusters around rapid/quick/swift-type intensifiers
- Another around innovation- and research-adjacent nouns
This is real specialisation, but it's at pattern-level, not domain-level. There's no "the translation expert" or "the grammar expert."
Mixtral 8x7B (2023), at LLM scale on general-purpose language modelling, found no such clean structure. That's the tension this section picks up: the specialisation debate isn't new, and it doesn't have a single settled answer.
There are at least three plausible explanations for why expert specialisation is still debatable.
- Scale changes what specialises. At LSTM-with-thousands-of-experts scale on a single task, fine-grained syntactic clusters are easy to detect. At LLM with hundreds of experts per layer scale on a much broader data distribution, that same structure might exist but be diluted across dramatically more traffic per expert.
- Task changes what specialises. Translation is a much narrower objective than general next-token prediction across a huge, heterogeneous corpus. Narrower tasks might simply produce cleaner, more legible specialisation.
- Methodology changes what you find. "Specialisation" isn't a single measurable quantity, how you probe for it (which tokens you test, which layer you look at, what counts as a "pattern") shapes what you're able to see.
5. DeepSeek-V3 and the Current State of the Art
DeepSeek-V3 represents the current high-water mark for this architecture, and it makes two specific bets that build directly on everything above.
Fine-grained experts + shared experts. Instead of a handful of large experts, DeepSeek-V3 uses many smaller experts per layer, alongside a small number of always-on "shared" experts that every token passes through regardless of what the gate decides.
Each MoE layer holds 256 routed experts plus 1 shared expert, and the gate selects the top 8 routed experts per token, so every token is processed by 9 experts total out of 257 available in that layer. The idea is to let the shared expert absorb general, broadly-useful knowledge, freeing the 256 routed experts to specialise on narrower, more distinctive patterns than they could if they also had to carry generic load.
This design choice isn't just architectural, DeepSeek ran empirical patterns to check if it actually does what it's supposed to. Turning off the shared expert causes a large capability drop, consistent with it having absorbed genuinely general-purpose knowledge. Turning off even a small slice of the top routed experts (as little as 1/16 of them) also meaningfully hurts performance; a real, measurable signal that the routed experts are holding specialised, non-redundant information. That's a useful data point to bring back into the specialisation debate from earlier: whatever these experts are specialising in, removing them provably costs the model something, even if we can't cleanly label what that something is.
Auxiliary-loss-free load balancing. Rather than the balancing loss described earlier, DeepSeek-V3 uses dynamically adjusted per-expert bias terms added to the gate's scores — nudging routing toward underused experts on the fly, without the accuracy cost that a traditional balancing loss can impose on the main training objective.
# Simplified DeepSeek-style routing sketch
def deepseek_moe_layer(x, routed_experts, shared_experts, gate_weights, bias, k):
logits = x @ gate_weights + bias # bias dynamically adjusted for balance
top_k_vals, top_k_idx = topk(logits, k)
top_k_weights = softmax(top_k_vals)
output = sum(s(x) for s in shared_experts) # always-on
for weight, idx in zip(top_k_weights, top_k_idx):
output += weight * routed_experts[idx](x) # sparse, routed
return output
The efficiency numbers. DeepSeek-V3 carries roughly 671 billion total parameters, but activates only around 37 billion per token. That gap is the whole MoE thesis made concrete: enormous total capacity, a fraction of it touched per token — and at 256 experts per layer, "a fraction" means well under 4% of any single layer's routed experts fire for a given token.
The engineering this demands. A pool this large only works with serious infrastructure underneath it: DeepSeek-V3 uses wide expert parallelism during training, spreading the 256 routed experts across many GPUs so each one handles a large enough batch of tokens to stay efficient, with token routing communicated over high-bandwidth interconnects between nodes. This is a direct, practical instance of the compute-vs-memory tension discussed later in this post, you can't just put 256 experts on one device and expect either training or serving to be fast.
6. Compute vs. Memory: MoE Solves One Bottleneck and Sharpens the Other
The "active params" compute gain is real, but it doesn't tell the whole story of how fast MoE models actually feel to use, because being compute or memory bound are two different bottlenecks, and MoE is genuinely lopsided in how it treats them.
Where MoE wins: compute-bound work. For any single token, only a handful of experts run, so the FLOPs cost per token scales with active parameters, not total parameters. This shows up most clearly during pre-fill, where the model processes the input prompt, typically with large batch sizes and math-heavy workloads. Pre-fill is compute-bound by nature, and cutting the required FLOPs is a direct, unambiguous win. It's also the main driver behind why training a 671B-parameter-class model can cost closer to what a much smaller dense model would.
Where MoE struggles: memory-bound work. The catch shows up on the other side of the ledger:
1) The full model still has to live in memory. Because any token can route to any expert, every expert's weights need to be resident in vRAM/RAM at all times.
2) Decode is naturally memory-bandwidth-bound, and MoE doesn't help. During decode, generating tokens one at a time, typically at small batch sizes, the bottleneck is moving weights from memory to the compute units, not doing arithmetic.
3) Dynamic, irregular routing adds its own overhead. Because each token can route to a different, unpredictable set of experts, fetching the relevant weights is less regular than in a dense model's fixed access pattern, which adds its own memory-traffic cost.
Who Actually Benefits — and Why Adoption Isn't a Clean Story
This split has a real consequence that doesn't get talked about enough: MoE's advantages and disadvantages land very differently depending on who's running the model.
1) Independent users and local/open-source deployments are usually running at small batch sizes, often a single user, one conversation at a time. That's exactly the decode-heavy, memory-bandwidth-bound regime where MoE's compute savings barely matter. A dense model of comparable active size can be a more practical fit here, since there's no giant idle parameter pool to store.
2) Large inference providers running frontier-scale models (1T+ total parameters) are the setting where MoE's tradeoffs actually pay off. At that scale, requests are batched heavily across many concurrent users, which pushes the workload back toward compute-bound territory and lets the FLOPs savings materialise as real throughput. Combined with expert parallelism across many GPUs, this is how labs serve enormous total-parameter models at low per-token cost and competitive latency.
So "is MoE worth it" doesn't have one answer. It depends heavily on batch size and deployment scale. It's a strong architecture for compute-bound, high-concurrency serving, and a comparatively weak fit for the memory-bound, low-batch regime that a lot of independent and local usage actually falls into.
7. Potential Research Areas
A few directions worth watching, stated with appropriate hedging since none of this is settled:
1) Deliberately steering specialisation - Biasing initialisation or auxiliary losses toward more domain-aware clustering, essentially a semi-supervised approach to routing, rather than leaving it fully emergent.
2) Routing stability across layers — does a token's full path through the network (its sequence of expert choices, layer by layer) carry any exploitable structure, or is each layer's decision genuinely independent of the others in any way that matters?
Conclusion
The through-line across three decades of this idea is more consistent than it might seem: a gating network learns to route inputs. Sparsity, introduced in 2017, turned MoE from a clever decomposition trick into a genuine solution to the compute-vs-capacity tradeoff, and that's the part of the story that's now settled and load-bearing in every frontier model.
What isn't settled is what the experts are. As expert counts keep climbing, understanding what's actually happening inside the routing may end up mattering as much as the routing mechanism itself.
Citations & Further Reading
- R. A. Jacobs, M. I. Jordan, S. J. Nowlan and G. E. Hinton, "Adaptive Mixtures of Local Experts," in Neural Computation, vol. 3, no. 1, pp. 79-87, March 1991, doi: 10.1162/neco.1991.3.1.79.
- Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017). . ArXiv. https://arxiv.org/abs/1701.06538
- Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., & Chen, Z. (2020). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. ArXiv. https://arxiv.org/abs/2006.16668
- Fedus, W., Zoph, B., & Shazeer, N. (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. ArXiv. https://arxiv.org/abs/2101.03961
- Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M. A., Stock, P., Subramanian, S., Yang, S., . . . Sayed, W. E. (2024). Mixtral of Experts. ArXiv. https://arxiv.org/abs/2401.04088
- Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y. K., Huang, P., Luo, F., Ruan, C., Sui, Z., & Liang, W. (2024). DeepSeekMoE: Towards Ultimate Expert Specialisation in Mixture-of-Experts Language Models. ArXiv. https://arxiv.org/abs/2401.06066
