Mixture of Experts

Here’s a riddle from the 2026 model landscape: a model can have hundreds of billions of parameters yet cost about the same to run as a model a fraction of its size. The trick is sparsity — on any given token, most of the model doesn’t fire. The mechanism is Mixture-of-Experts (MoE), and it’s why models like Mixtral, DeepSeek, and Qwen-MoE can be enormous on paper and still affordable to serve.

One big FFN → many small experts

Recall from the transformer block that the feed-forward (FFN) sublayer holds most of a transformer’s parameters and runs on every token. MoE replaces that single FFN with N smaller expert FFNs plus a tiny router. For each token, the router scores the experts and sends the token only to its top-k (usually k=1 or 2). The other experts are skipped entirely for that token.

So the model stores all N experts, but any single token only computes k of them. Here one token is routed through an 8-expert layer with top-2 routing — six experts never fire for it:

The bargain: capacity scales with N, compute with k

That’s the whole point, stated as a tradeoff:

• Parameters (capacity, “knowledge”) grow with the number of experts N.
• Compute per token (FLOPs, cost) grows only with k, the experts actually used.

A model with 8 experts and top-2 routing has roughly the parameter count of 8 FFNs but the per-token compute of 2. You get a much larger, more capable model without paying to run all of it on every token. This is why “total parameters” and “active parameters” are now reported separately — a model might be “236B total, 21B active.”

The catch: load balancing

A naive router has a failure mode: it learns to love a few experts and send everything to them, while other experts get no tokens, no gradient, and never learn. That wastes most of the model’s capacity — the per-expert load ends up wildly uneven.

The standard fix is an auxiliary load-balancing loss added during training that penalizes imbalance, nudging the router to spread tokens evenly across experts. Here’s the imbalance, made concrete:

import numpy as np

rng = np.random.default_rng(1)
n_experts, n_tokens, k = 8, 2000, 2

# A biased router: experts 0-1 are over-preferred (the failure mode).
bias = np.array([3.0, 2.5] + [0.0]*(n_experts-2))
load = np.zeros(n_experts, dtype=int)
for _ in range(n_tokens):
    scores = rng.standard_normal(n_experts) + bias
    topk = np.argpartition(scores, -k)[-k:]      # pick top-k experts
    load[topk] += 1

ideal = n_tokens * k / n_experts
print("tokens routed to each expert:", load.tolist())
print(f"ideal (balanced) per expert: {ideal:.0f}")
print(f"busiest/idle ratio: {load.max() / max(load.min(),1):.1f}x  "
      f"-> needs a load-balancing loss")

tokens routed to each expert: [1876, 1713, 64, 75, 63, 71, 62, 76]
ideal (balanced) per expert: 500
busiest/idle ratio: 30.3x  -> needs a load-balancing loss

The two favoured experts swallow ~3,600 of the 4,000 routed slots while the other six starve — a 30× imbalance. The auxiliary loss is what pushes those bars back toward the ideal of 500 each, so every expert sees enough tokens to actually learn.

In one breath

• Mixture-of-Experts replaces the single dense FFN with N smaller expert FFNs plus a tiny router that sends each token to only its top-k (usually 1–2).
• The bargain: capacity (parameters) scales with N, compute (FLOPs) scales only with k — which is why models now report “236B total, 21B active.”
• It saves compute, not memory: all N experts still sit in VRAM, so the serving bottleneck shifts from FLOPs to memory and to networking tokens to the right expert.
• A naive router collapses onto a few favourite experts (a 30× load imbalance in the demo), starving the rest — so MoE training adds an auxiliary load-balancing loss to spread tokens evenly. This economics is why most 2026 frontier/open models (Mixtral, DeepSeek, Qwen-MoE) are sparse.

Quick check

Next

That completes the Deep Learning track’s modern-architecture arc. From here, the Generative AI track takes these building blocks into production — serving, RAG, reasoning models, and evaluation.