An MoE model advertises much lower active-parameter cost than its total parameter count. What does that mean operationally, and how would expert imbalance, routing overhead, expert parallelism, and a hot expert affect throughput and tail latency?

How to think about it

An MoE model’s active-parameter count is the amount of expert computation a token actually invokes, while its total count includes every expert stored in the model. Operationally, that can lower arithmetic per token without lowering weight memory, routing communication, or latency: the slowest, busiest expert and the dispatch path often set the real result.

Why the numbers differ

A Mixture-of-Experts model replaces some dense layers, usually the feed-forward network, with a pool of separate expert networks. A router, also called a gate, looks at each token’s hidden representation and chooses which experts should process it.

If the model has eight experts and uses top-2 routing, each token visits two experts rather than all eight. The total parameter count includes all eight expert networks. The active count includes only the selected two, plus any shared layers such as attention.

This is conditional computation: different tokens pay for different parts of the model.

Suppose a model has:

• Eight experts, each with 1 billion parameters
• Shared layers containing 2 billion parameters
• top-2 routing

The model has roughly 10 billion parameters in total. A token activates about 2B + 2B = 4B parameters along its path: 2 billion in shared layers and 2 billion across its two selected experts.

A dense 10-billion-parameter model would execute its full feed-forward path for every token. The MoE model may therefore need much less arithmetic per token.

But “active parameters” is not the same thing as “memory required” or “wall-clock cost.” Unless the system offloads weights, all 8 billion expert parameters still need to live somewhere: GPU memory, host memory, or another machine. The model has made computation sparse, not magically made the unused weights disappear.

Quantity	What it tells you	What it does not tell you
Total parameters	How many weights exist	Per-token arithmetic or latency
Active parameters	Which weights one token uses	Memory footprint or communication cost
Step time	How long the batch actually takes	Whether every token received equal compute

Parameter count is a useful proxy for arithmetic, especially for comparing similar architectures. It is not an invoice.

The concrete batch problem

Take 4,096 input tokens. With top-2 routing, the system creates 8,192 expert assignments. With eight experts, perfectly even routing would send 1,024 assignments to each expert.

Now imagine the actual distribution is:

• Expert 1: 1,800 assignments
• Expert 2: 400 assignments
• The other six: about 999 each

The average is still 1,024. A dashboard showing only average tokens per expert might look healthy. Expert 1, however, has about 1.76 times the average compute load. If compute dominates and the experts are otherwise identical, that expert’s device needs roughly 1.76 times as much expert work.

That is expert imbalance: tokens are not spread evenly across experts. The batch finishes when the slowest expert finishes, so the useful capacity is closer to the maximum load than the average load.

Many implementations impose a capacity limit per expert. If the illustrative capacity is 1,280 assignments, Expert 1 has 520 assignments beyond its slots. Depending on the implementation, those assignments may be dropped, sent to another selected expert, handled by a fallback path, or padded and processed differently. Each choice has a cost. Dropping can hurt quality. Rerouting changes the intended computation. Padding wastes compute.

A balancing loss during training encourages the router to use experts evenly. It helps, but it is not a production guarantee. A model trained on a broad mixture of data can still see a serving batch dominated by code, legal text, or one language. The router may quite reasonably send many of those tokens to the same expert.

Routing overhead is real work

The router itself first computes scores for every token and expert, then selects the top choices. The system must also:

1. Group tokens by destination expert.
2. Move or copy their hidden states into expert-sized batches.
3. Run the expert networks.
4. Scatter the results back to the original token order.
5. Combine the outputs when a token used multiple experts.

That work is often called dispatch and combine. It involves indexing, permutation, memory movement, kernel launches, and sometimes synchronization.

With expert parallelism, it also involves network traffic. top-2 sends two expert assignments per token, so it creates roughly twice the expert-destination traffic of top-1. That does not mean exactly twice the total latency, because compute and message sizes matter, but it is plainly not free.

This overhead behaves differently during prefill and decoding. During prefill, a request supplies many tokens at once, so routing and communication can be amortized over larger expert batches. During autoregressive decoding, a request contributes roughly one new token per step. Small batches leave less work to amortize fixed dispatch and collective-communication costs. A model advertised as “4 billion active parameters” can therefore lose to a smaller dense model on low-batch, latency-sensitive traffic.

What expert parallelism changes

Expert parallelism, or EP, shards experts across devices: each device owns some experts rather than holding every expert copy. It solves two practical problems.

First, the full expert pool can exceed the memory of one device. Second, several experts can process different token groups simultaneously.

The price is an all-to-all exchange inside the expert-parallel group. A device sends each token activation to the device that owns its selected expert, waits for the expert result, and then receives the result back. If all eight experts are spread across eight GPUs, a token routed to Expert 3 may leave the GPU where it entered and travel to the GPU hosting Expert 3.

Uniform routing makes this arrangement efficient. Each device receives approximately the same amount of work, and the interconnect carries a predictable amount of traffic.

Imbalanced routing makes it ugly. One device receives a large queue while the others sit partly idle. The collective still has to coordinate, and the MoE layer usually cannot move on until the required outputs are available. Fast GPUs do not rescue a slow rank when the batch is synchronized around that rank.

EP can improve aggregate throughput when batches are large and the GPUs have a fast interconnect. It can hurt single-request latency when tokens cross machines or when expert batches are too small to use efficient kernels. The hardware topology matters. Eight experts on eight GPUs connected within one server is a different system from eight experts scattered across four hosts.

A hot expert and the tail

A hot expert is an expert that is persistently selected far more often than its peers across many batches, not merely one unlucky batch. Causes include router bias, a narrow traffic mix, or an expert becoming especially attractive for common patterns.

The first symptoms are operational:

• One GPU or rank has consistently higher utilization.
• That rank’s expert queue grows.
• All-to-all operations show uneven send or receive volumes.
• Average throughput falls even though several GPUs are idle.
• p50 latency may look acceptable while p95 or p99 latency climbs.

The tail gets worse because generation is stepwise. If one sequence in a batch routes its next token to a congested expert, the batch may wait for that expert before producing the next token. In continuous batching, that delay can also hold up other requests sharing the scheduling cycle.

Typical mitigations include a stronger load-balancing objective, routing jitter, a capacity policy, and replicating a hot expert on multiple devices. Replication reduces the queue by giving the router more destinations, but it consumes extra memory and may increase synchronization and routing complexity. Forcing perfectly uniform routing can also send a token to a less suitable expert and reduce model quality. There is no free “balance” switch.

The senior answer is therefore not “MoE is cheap.” It is “MoE trades dense arithmetic for sparse arithmetic plus a distributed scheduling problem.”

What they’ll ask next

Does a lower active count reduce model memory?
Usually not. The full expert pool must still be resident somewhere unless the serving system uses weight offload or paging. Active parameters primarily describe per-token computation.

How would you diagnose poor MoE performance?
Measure tokens per expert per layer, dispatch and combine time, all-to-all bytes and duration, expert queue time, dropped or rerouted tokens, GPU utilization by rank, and request latency percentiles. Average GPU utilization alone can hide one overloaded expert.

How would you fix a hot expert?
First separate transient batch imbalance from persistent skew. Then consider better balancing during training, capacity and routing policies, expert replication, and a serving topology that keeps expert traffic on fast links. Validate both quality and p99 latency; a fix that balances GPUs by degrading routing quality is not a complete fix.

One line to say in the room: “Active parameters reduce the arithmetic each token invokes, but real MoE throughput and p99 are governed by routing, interconnect traffic, and the most heavily loaded expert—not by the average active count.”