Frontier LLM walkthrough: DeepSeek-V3

By now you have met the parts of a modern LLM one at a time: pre-norm with RMSNorm, RoPE, the GQA/MLA attention ladder, SwiGLU, Mixture of Experts, multi-token prediction, and the training-side tricks of mixed precision and pipeline parallelism. A real frontier model is not a new idea on top of these — it is the assembly of them into one stack where every choice is made to buy capability per dollar. DeepSeek-V3 is the cleanest worked example, because its team documented the whole recipe.

The stack, top to bottom

DeepSeek-V3 is a 671-billion-parameter Mixture-of-Experts decoder, of which only about 37 billion are active on any given token. Here is the decoder block, with each piece linked to the lesson that explains it:

Reading the choices

Each component on that diagram is a deliberate cost trade:

• MoE (671B total / 37B active). Mixture of Experts stores a huge parameter count for capacity but routes each token to only a few experts, so per-token compute stays small. DeepSeek-V3 adds auxiliary-loss-free balancing — it nudges routing with a learned per-expert bias instead of an auxiliary loss, avoiding the small quality tax that the load-balancing loss usually imposes.
• MLA — Multi-head Latent Attention. The newest rung of the attention ladder. Instead of caching full keys and values per head (huge) or sharing them across heads (GQA), MLA compresses K and V into a small shared latent vector and reconstructs them on the fly. The KV cache shrinks dramatically, which is what makes long-context inference cheap.
• MTP — Multi-token prediction. An extra head predicts the second future token, giving a denser training signal and a built-in draft model for self-speculative decoding at serving time.
• FP8 training. Most of the matmuls run in 8-bit floating point — cheaper and faster than bf16 — with careful scaling to stay numerically safe (the precision story from mixed precision, pushed one step lower).
• DualPipe. The pipeline-parallel schedule that overlaps communication with computation to keep thousands of GPUs busy.
• The familiar base: pre-RMSNorm, RoPE, SwiGLU experts — the modern-LLM defaults.

The one number that captures the philosophy: a model with the capacity of 671B parameters that costs about a 37B model to run per token.

total_params_B  = 671     # parameters stored (capacity)
active_params_B = 37      # parameters used per token (MoE top-k routing)

print(f"capacity : {total_params_B}B parameters stored")
print(f"compute  : {active_params_B}B active per token = {active_params_B / total_params_B:.1%}")
print(f"-> a {total_params_B}B-capacity model that runs at ~a {active_params_B}B model's cost")

capacity : 671B parameters stored
compute  : 37B active per token = 5.5%
-> a 671B-capacity model that runs at ~a 37B model's cost

The other branch: hybrid SSM models

Not every frontier model is pure attention. Jamba mixes three ingredients: mostly Mamba state-space-model layers (linear-time sequence mixing, a constant-size recurrent state, no growing KV cache), a few interleaved attention layers (for the exact long-range recall that SSMs lack), and MoE (for cheap capacity). The result keeps most layers linear and cheap while retaining attention’s precision where it matters — a different way to reach the same goal of long context at low cost.

So the frontier splits into two families that borrow from each other: attention-centric stacks like DeepSeek-V3 that lean on MoE + MLA, and hybrid SSM stacks like Jamba that lean on Mamba + a little attention. Both are assemblies of the same toolkit, tuned for capability per dollar.

In one breath

• A frontier LLM is the assembly of the chapter’s parts into one cost-optimized stack — not a new idea on top of them.
• DeepSeek-V3: a 671B-total / 37B-active MoE decoder (auxiliary-loss-free routing), MLA (compressed KV latent → tiny cache, cheap long context), MTP (denser signal + self-speculative decode), trained in FP8 with DualPipe, on the usual pre-RMSNorm / RoPE / SwiGLU base.
• The philosophy in one number: 5.5% of parameters active per token — 671B of capacity at roughly a 37B model’s running cost.
• Each choice decouples a cost from a capability (capacity from compute, context from cache, throughput from one-token-per-pass) and they compound.
• The other branch is hybrid SSM models like Jamba (mostly Mamba + a few attention layers + MoE) — same goal, different toolkit mix.

Quick check

Next

That closes the architecture arc — from a single attention head to a full frontier stack. To go up a level into how these models are used in production — serving, RAG, evaluation, and cost — head into the Generative AI track.