Pipeline parallelism

Data parallelism (DDP) puts a full copy of the model on every GPU and splits the data — which is useless the moment the model itself no longer fits on one GPU. A frontier model’s weights, gradients, and optimizer state run to terabytes; you have to split the model. Pipeline parallelism is the most intuitive way to do it: cut the network into consecutive stages and put each stage on a different GPU, like an assembly line.

GPU 0 holds the first chunk of layers, GPU 1 the next, and so on. A batch flows through stage 0, whose output is handed to stage 1, then stage 2… That works — but done naively it wastes almost everything.

The pipeline bubble

Run a single batch through the line and watch the GPUs: while stage 0 computes, stages 1–3 sit idle waiting for its output. When the work reaches stage 1, stage 0 has nothing to do. At any instant only one stage is busy — so with P stages you get 1/P utilization. Four GPUs running at 25% is a catastrophic way to spend a cluster. That idle time is the pipeline bubble.

The fix is to keep all stages busy at once by feeding the pipeline a stream of work instead of one batch. Split the batch into M micro-batches and pump them in back-to-back (this is GPipe): as micro-batch 1 advances to stage 1, micro-batch 2 enters stage 0 behind it, and soon every stage is working on a different micro-batch.

How much bubble is left?

The bubble never fully vanishes — you still pay one ramp-up and one ramp-down — but its fraction shrinks as you add micro-batches. For P stages and M micro-batches the bubble fraction is (P − 1) / (M + P − 1):

def bubble_fraction(P, M):
    return (P - 1) / (M + P - 1)

P = 4  # four pipeline stages
for M in [1, 2, 4, 8, 16, 32]:
    print(f"micro-batches M={M:>2}: bubble = {bubble_fraction(P, M):.1%} idle")

micro-batches M= 1: bubble = 75.0% idle
micro-batches M= 2: bubble = 60.0% idle
micro-batches M= 4: bubble = 42.9% idle
micro-batches M= 8: bubble = 27.3% idle
micro-batches M=16: bubble = 15.8% idle
micro-batches M=32: bubble = 8.6% idle

M = 1 is the naive case — 75% of a 4-GPU cluster idle. Push 32 micro-batches through and the bubble falls under 10%. The rule of thumb: keep M several times larger than P. You can’t shrink it forever, though — more micro-batches means more in-flight activations to hold in memory, which is the next problem.

1F1B and DualPipe

1F1B (“one-forward-one-backward”, from PipeDream) tackles that memory cost. Plain GPipe runs all M forward passes, then all M backward passes — so it must store activations for every in-flight micro-batch at the peak. 1F1B instead interleaves: as soon as a micro-batch finishes its forward pass through the last stage, it starts flowing backward, freeing its activations early. Same bubble, far lower peak activation memory — which pairs naturally with activation checkpointing.

DualPipe (DeepSeek-V3) goes further: it runs the pipeline bidirectionally and deliberately overlaps the inter-stage communication with computation, so the GPU is computing one micro-batch while it ships another’s activations across the network. That overlap hides almost all of the remaining bubble — a big part of how DeepSeek-V3 trained so cheaply.

In one breath

• When a model is too big for one GPU, data parallelism can’t help (it replicates the whole model); you must split the model, and pipeline parallelism splits it by layer into stages, one per GPU — an assembly line.
• Naively only one stage works at a time (1/P utilization); the idle time is the pipeline bubble.
• GPipe micro-batching streams M micro-batches so all stages stay busy; the bubble fraction is (P−1)/(M+P−1) — ~75% at M=1 but under 10% at M=32 for 4 stages.
• 1F1B interleaves forward/backward to cut peak activation memory; DualPipe (DeepSeek-V3) overlaps communication with compute to hide almost all of the remaining bubble.
• Pipeline is one axis of 4D parallelism — data, tensor, pipeline, and expert splits combined to train the largest models.

Quick check

Next

Pipeline parallelism is one piece of how frontier models are trained; see how it and the rest of the modern stack come together in the frontier LLM walkthrough, and revisit Multi-GPU training for the data-parallel half (DDP & FSDP).