What is Mixture of Experts (MoE) and how does it improve LLM scalability?
MoE uses a learned router to send each token to a small number of expert feed-forward networks instead of one feed-forward network shared by every token. It increases total model capacity without making per-token expert computation grow with the number of experts, although memory, routing communication, and load balancing become harder.
How to think about it
An MoE LLM is a Transformer model whose feed-forward sublayer, the token-wise multilayer perceptron after attention, contains multiple expert networks. A learned router sends each token to only a small number of those experts. This lets the model add a great deal of total parameter capacity without making every token pay for every expert.
Why MoE improves scalability
A conventional dense Transformer uses the same feed-forward network for every token. If that network has 10 billion parameters, every token passes through the relevant matrix multiplications for those 10 billion parameters. Making the model wider increases both its capacity and its compute cost.
MoE separates those two costs.
Suppose an MoE layer has N experts. Each expert is usually an independent feed-forward network with the same input and output dimensions. For each token, a router examines the token’s hidden representation, produces one score per expert, and selects the best k experts. Common choices are top-1 and top-2 routing.
Conceptually, for a token representation h, the router computes a score vector such as softmax(W_r h). It keeps the top k scores, sends h to those experts, and combines their outputs using the selected scores. The other experts do not run for that token.
That is the key mechanism: the number of available experts can grow while the number activated for one token stays fixed.
The attention part of the Transformer is usually still dense. The sparsity normally applies to the feed-forward sublayer, which is large enough to make the trade-off worthwhile. Also, an expert is not a separate full LLM. It is usually one MLP inside one Transformer layer. The same token can be routed to different experts in different layers.
Experts may develop useful specializations, but this is not guaranteed. One expert might become better at code-like patterns and another at certain languages, but the model is not explicitly told to create a “math expert” or a “French expert.” The router and training data determine what specialization emerges.
A numerical example
Take a conventional feed-forward layer with an input size of 4,096 and an intermediate size of 16,384. It projects from 4,096 to 16,384 and then back again. Ignoring biases, its parameter count is:
4,096 × 16,384 + 16,384 × 4,096 = 134,217,728
That is about 134 million parameters.
Now replace that layer with 16 experts of the same size:
- Total expert parameters: about 2.147 billion
- Experts selected per token: 2
- Expert parameters applied to one token: about 268 million
So the layer has 16 times the expert capacity of the original dense layer, but a token only uses two experts. In this simplified comparison, the token pays for roughly two feed-forward networks rather than sixteen.
Compared with the original single dense layer, top-2 routing costs about twice the expert computation. That is not “free.” But compared with a dense layer containing all 16 copies, it uses about one-eighth of the expert computation per token.
This is why MoE papers and model cards often distinguish total parameters from active parameters. Total parameters describe all weights stored in the model. Active parameters describe the weights used for one token, usually including some convention about shared layers. A model advertised as having 1 trillion total parameters and 100 billion active parameters does not have the memory footprint of a 100-billion-parameter dense model. The unused experts still have to live somewhere.
What happens in a real system
During training, a batch may contain thousands of tokens. The router assigns each token to its selected experts. The system then groups tokens by destination expert, runs each expert’s matrix multiplications, and puts the outputs back into their original token positions.
When experts are spread across accelerators, this is called expert parallelism: different devices own different experts. If 64 experts are evenly distributed across 8 GPUs, each GPU might own 8 experts. But a token on GPU 1 may be routed to an expert on GPU 6. Its activation must cross the interconnect.
That movement is usually implemented as an all-to-all operation, where every device sends different token slices to other devices and later receives the processed results. With a fast intra-node link, this can be manageable. Across machines, network bandwidth and synchronization can become the bottleneck.
This creates a practical rule: MoE tends to look better at large batch sizes and high inference concurrency. More tokens give each expert enough work to keep its accelerator busy and help amortize communication. At low traffic, the system may spend more time sorting, dispatching, synchronizing, and waiting than doing useful matrix multiplication. A dense model can then be faster and simpler, even if it has fewer parameters.
The hard part: load balancing
A router can make a locally sensible decision for each token while making a globally terrible decision for the system. If it sends too many tokens to one expert, that expert becomes overloaded while other GPUs sit idle.
For example, consider 10,000 tokens, 16 experts, and top-2 routing. There are 20,000 expert assignments in total. Perfectly even routing would give each expert 1,250 assignments. If one expert receives 5,000 assignments, it becomes the queue everyone waits for.
MoE systems therefore impose a capacity limit. With a capacity factor of 1.25, an expert’s nominal 1,250 slots would become roughly 1,563 slots, allowing some imbalance without allocating space for the worst possible routing pattern. The exact rounding and overflow behavior depends on the implementation.
When an expert exceeds capacity, the extra token may be dropped from the expert computation, sent to a fallback expert, or handled through a residual or shared path. Dropping tokens sounds alarming because it is alarming: enough overflow can hurt quality. Some systems use dropless routing, but that shifts the cost into scheduling, memory, or communication.
Training commonly adds an auxiliary load-balancing loss. It encourages the router’s probabilities and actual token assignments to spread across experts. The trade-off is real. Push too hard for equal routing and the router may lose useful specialization; push too little and one expert can collapse into a hot spot.
A useful production dashboard tracks per-layer expert counts, overflow or dropped-token rate, router statistics, per-device utilization, and all-to-all time. Average GPU utilization can hide the problem. One GPU at 100 percent and seven GPUs waiting is not healthy parallelism.
The senior-level trade-off
MoE improves the capacity-versus-compute trade-off. It does not reduce every resource cost.
| Resource | Dense model | MoE model |
|---|---|---|
| Stored expert weights | One network | All experts |
| Expert compute per token | All dense weights | Selected experts |
| Routing overhead | Minimal | Router and dispatch |
| Cross-device traffic | Usually lower | Often significant |
| Failure risk | Simpler | Imbalance and overflow |
MoE is attractive when the target model needs more capacity than one accelerator can compute densely at an acceptable cost. It is less attractive when the workload has tiny batches, strict tail-latency requirements, weak interconnects, or a small deployment budget.
Top-1 routing minimizes compute and communication. Top-2 routing costs more but can let two experts contribute and may improve quality or training behavior. There is no universally correct k; it depends on model size, hardware, batch size, and the quality target.
Some modern designs also add shared experts that process every token alongside routed experts. That can preserve common knowledge while the routed experts handle conditional capacity. It also means the advertised active-parameter count must be read carefully.
A failure mode you would see first
The first symptom of router collapse is often operational rather than mathematical: one expert device is pegged, other devices are lightly used, p99 latency rises, and overflow counters spike. During training, the loss may become noisy or stop improving.
The likely causes include an imbalanced router, insufficient capacity, a batch that is too small, or a network bottleneck mistaken for a compute problem. Inspect expert assignment histograms and dispatch time before changing the model. Adding more experts does not fix a router that already prefers one expert.
What they’ll ask next
Does MoE reduce memory usage?
No, not automatically. All expert weights must be stored somewhere. Expert parallelism divides that memory across devices, so each GPU holds fewer weights, but aggregate memory remains large. MoE reduces computation relative to a dense model with the same total expert capacity; it does not make the total weights disappear.
Why not always use top-1 routing?
Top-1 is cheaper because each token visits one expert. Top-2 can improve the quality trade-off by combining two expert outputs and giving the router more flexibility, but it doubles the expert-side work and usually increases communication. The right choice comes from measuring quality, throughput, memory, and tail latency together.
How do you prevent one expert from receiving all the tokens?
Use a load-balancing objective, per-expert capacity limits, and careful monitoring of assignment counts and overflow. Router noise or regularization can help during training, and dropless or fallback routing can protect quality. The important point is to balance both the router’s probabilities and the actual token counts; a router that looks balanced on paper can still create uneven device work.
Say this in the interview: MoE scales LLM capacity by storing many expert feed-forward networks but routing each token through only a few, trading dense compute for memory, communication, and load-balancing complexity.