What is the difference between DDP and FSDP for distributed training?
DDP keeps a complete model, gradient set, and optimizer state on every GPU and synchronizes gradients, while FSDP shards those states across GPUs and temporarily gathers the parameters needed by each wrapped module. DDP is simpler and often faster when the model fits; FSDP trades more communication and checkpointing complexity for much lower per-GPU memory.
How to think about it
The short answer
DDP, or Distributed Data Parallel, keeps a complete copy of the model on every GPU and synchronizes gradients after each backward pass. FSDP, or Fully Sharded Data Parallel, shards the model’s parameters, gradients, and optimizer state across GPUs, then temporarily gathers the parameters needed for each module.
DDP is simpler and often faster when the complete training state fits on one GPU. FSDP uses more communication and has more operational complexity, but its lower per-GPU memory lets you train models that DDP cannot fit.
What DDP actually does
Suppose four GPUs each run one worker process, called a rank. DDP gives every rank the complete model, then gives each rank a different slice of the training batch.
Each rank performs the forward pass on its own data. During backpropagation, it computes a gradient, the signal that tells the optimizer how each trainable model number should change. DDP then performs an all-reduce, a communication operation that combines the corresponding gradients from every rank and gives the combined result back to all ranks. In the usual setup, that combination is an average.
So if one GPU computes a gradient of 2 for a parameter and the other three compute 4, 6, and 8, every GPU receives the average, 5. Every rank then applies the same optimizer update. The model copies remain identical.
DDP usually all-reduces gradients in buckets while backpropagation is still running. It does not necessarily wait for every layer to finish before communicating. That overlap is one reason DDP can be quite efficient.
DDP is data parallelism, not model parallelism. The batch is split across GPUs; the layers are not. Every GPU needs the full model, its gradients, and normally its own copy of the optimizer state. Optimizer state means extra tensors maintained by the optimizer, such as Adam’s running first- and second-moment estimates.
There is a small assumption hiding here: the ranks must use compatible loss scaling and, normally, equal local batch sizes. DDP synchronizes numbers. It does not automatically make an uneven sampling scheme statistically correct.
Why FSDP exists
Memory is usually the reason to choose FSDP.
Consider a model with 7 billion trainable parameters. In bfloat16, a 2-byte number format, the parameters alone require about 14 GB. The gradients require another 14 GB. A common Adam setup keeps two FP32, 4-byte, moment tensors per parameter:
- Parameters: 7 billion times 2 bytes = about 14 GB
- Gradients: 7 billion times 2 bytes = about 14 GB
- Adam moments: 7 billion times 2 tensors times 4 bytes = about 56 GB
That is already about 84 GB per GPU, before activations, temporary buffers, CUDA overhead, or a possible FP32 master copy of the weights. DDP therefore cannot place this training state on an 80 GB GPU, even if you have four such GPUs. It replicates the same roughly 84 GB on every GPU.
With the usual full-shard FSDP configuration, the large tensors are divided across the four ranks. The idealized persistent-state calculation becomes roughly 21 GB per GPU instead of 84 GB. That is the central win.
| Concern | DDP | FSDP |
|---|---|---|
| Parameters | Full copy per GPU | One shard per GPU between uses |
| Gradients | Full copy per GPU | One shard per GPU |
| Optimizer state | Full copy per GPU | Sharded across GPUs |
| Main communication | Gradient all-reduce | Parameter all-gather and gradient reduce-scatter |
| Main advantage | Simplicity and speed | Much lower per-GPU memory |
The saving is not compression. The total number of model-state bytes across the cluster is roughly similar. FSDP changes where those bytes live.
What happens during an FSDP step
FSDP divides the model into FSDP units, which are wrapped modules chosen as communication and memory boundaries. For a transformer, a common design is to wrap individual transformer blocks rather than the entire language model.
With full sharding, a rank normally keeps only its own parameter shard between computations. Before a wrapped block runs, FSDP performs an all-gather: ranks exchange their shards so every rank temporarily has the block’s complete parameters. The rank computes the block’s forward pass, and FSDP releases the parameter pieces that rank does not own.
During backward, FSDP gathers the parameters again when needed. It then uses reduce-scatter, which combines corresponding gradients across ranks while leaving each rank with only its portion of the result. The optimizer updates that local shard.
The important misconception is that FSDP does not make rank one compute only one-quarter of a transformer block. Each rank generally computes the complete wrapped block on its own data. FSDP mainly shards persistent storage and pays communication costs to reconstruct what computation needs.
Wrapping granularity matters. Wrapping the entire 7-billion-parameter model as one unit can force a giant all-gather and erase much of the memory benefit. Wrapping smaller units lowers the peak memory needed for temporary full parameters, but creates more communication events.
The nuance that earns the senior signal
FSDP does not solve every memory problem. It primarily reduces memory for parameters, gradients, and optimizer state. Activations, the intermediate results saved for backpropagation, can still dominate memory for long sequences or large batches.
Activation checkpointing reduces that cost by saving fewer intermediate results and recomputing them during backward. Smaller per-GPU batches, shorter sequences, and gradient accumulation can also help. If the model fits its weights under FSDP but still fails during a long-sequence forward pass, sharding was not the missing piece.
FSDP also communicates more frequently than DDP. DDP mainly synchronizes gradients. FSDP must gather parameters for computation and scatter gradients afterward. Fast NVLink inside one machine can make that trade-off reasonable. A slower multi-node network can turn the extra collectives into the dominant part of step time.
Topology affects the choice. A hybrid sharding strategy can shard within a node while replicating between nodes, reducing expensive cross-node traffic. The right choice depends on GPU memory, interconnect bandwidth, model size, sequence length, and how much computation exists to overlap with communication.
DDP is often the better engineering choice when the model fits with meaningful headroom. It has a simpler mental model, simpler debugging, and usually simpler checkpoint handling. FSDP introduces additional concerns around wrapping, initialization, optimizer-state conversion, and checkpoint formats. A full checkpoint may require gathering sharded state, while sharded checkpoints preserve the memory advantage.
FSDP also has a hard boundary: the complete model must fit across the aggregate available memory, and each FSDP unit must fit on a rank when its parameters are temporarily gathered. If one indivisible unit is larger than a GPU, FSDP alone cannot fix it. Tensor parallelism or pipeline parallelism may then be needed, often alongside FSDP.
A failure mode you would actually see
A common FSDP failure looks like this: nvidia-smi shows only 24 GB used per GPU, then the job reports CUDA out of memory when the first large transformer block begins.
The usual cause is a coarse FSDP unit. Its all-gather temporarily materializes too many full parameters, and activations are allocated on top. Finer-grained wrapping, activation checkpointing, or a smaller microbatch addresses different parts of that peak.
Another practical failure is a collective timeout at the first all-gather. Every rank must execute the same FSDP collectives in the same order. A rank that skips a batch, takes a different control-flow branch, or initializes a differently wrapped model can leave the other ranks waiting indefinitely.
What they’ll ask next
Does FSDP eliminate communication?
No. It changes the communication pattern. DDP communicates gradients, while FSDP communicates parameter shards before computation and gradient shards after computation. FSDP can still be faster overall when the alternative is running out of memory, but it is not a free memory reduction.
Can FSDP train a model larger than one GPU?
Yes, provided the model state fits across the available GPUs and each wrapped unit, its temporary full parameters, and its activations fit on one GPU. It does not make an arbitrarily large individual layer fit. Very large systems often combine FSDP with tensor or pipeline parallelism.
When would you choose DDP instead?
I would choose DDP when the model, optimizer state, gradients, and activations fit comfortably on each GPU. I would choose FSDP when replicated model state is the memory bottleneck and the cluster has enough communication bandwidth to tolerate parameter all-gathers and gradient reduce-scatters.
Say this in the interview
“DDP replicates the full training state and synchronizes gradients, whereas FSDP shards parameters, gradients, and optimizer state and gathers only the module needed for computation; I use DDP for simplicity when the model fits, and FSDP when per-GPU memory is the limiting constraint.”