datarekha
Deep Learning Medium

What is gradient accumulation and why is it useful?

For ML EngineerMLOps Engineerresearch-engineer
The short answer

Gradient accumulation runs several forward and backward passes without zeroing gradients, sums them, and only steps the optimizer after N micro-batches, simulating a larger effective batch size than fits in memory. It lets you train with large effective batches on limited GPU memory at the cost of more compute per update.

How to think about it

Gradient accumulation runs several forward and backward passes without zeroing gradients, sums them, and only steps the optimizer after N micro-batches, simulating a larger effective batch size than fits in memory. It lets you train with large effective batches on limited GPU memory at the cost of more compute per update.

Learn it properly Batch size ↔ learning rate

Keep practising

What is gradient accumulation and when do you need it? How does batch size affect training — speed, convergence, and generalisation? What is gradient clipping and when would you use it? In a PyTorch training loop, why do you need to call optimizer.zero_grad() before backpropagation? What is gradient clipping, and when is it necessary?
All Deep Learning questions

Explore further

Gradient descent The training loop Vanishing & exploding gradients
Batch Size Exploding Gradient Gradient Clipping Vanishing Gradient
Skip to content