What is gradient accumulation and when do you need it?

How to think about it

Gradient accumulation runs several small micro-batches, adds their gradients, and calls optimizer.step() only after the group is complete. You need it when the batch size you want does not fit in GPU memory; it saves activation memory, but it does not make training faster by itself.

Why it works

A micro-batch is the small batch processed in one forward and backward pass. An optimizer step is the operation that uses the gradients to update the model weights and the optimizer’s internal state.

Normally the loop is:

1. Take a batch.
2. Run the forward pass.
3. Compute the loss.
4. Run backpropagation.
5. Update the weights.
6. Clear the gradients.

With accumulation, steps 1 through 4 happen several times. The weights remain unchanged. Gradients are added to the parameter gradient buffers. Only then do you update the weights and clear the buffers.

Why bother? A larger batch gives the optimizer an average gradient based on more examples. Individual examples often pull the weights in slightly different directions. Averaging 32 examples cancels some of that random variation, producing a less noisy update. That can make training more stable and sometimes lets you use a larger learning rate.

The key word is average.

Suppose three micro-batches produce a scalar gradient of 2, 6, and 4. The gradient for a true batch containing all three is:

(2 + 6 + 4) / 3 = 4

If you simply add the gradients without dividing, you get 12. With stochastic gradient descent and a learning rate of 0.01, the parameter change is three times larger than intended. For adaptive optimizers such as Adam, the effect is more complicated because the optimizer normalises gradients, but the gradient scale still affects clipping, numerical stability, epsilon terms, and optimizer state. It is still the wrong calculation.

The usual implementation divides the loss by the number of accumulation steps before calling backward().

Memory is the other half of the mechanism. During the forward pass, the model stores activations, intermediate values needed to calculate gradients. A large batch needs activations for every example at once. With accumulation, the model only needs activations for one micro-batch. After its backward pass, that computation graph can usually be released. The parameter gradients remain, but they are much smaller than the full activation set.

Gradient accumulation therefore reduces peak activation memory. It does not reduce the memory required for model weights, parameter gradients, or optimizer state. If the model and Adam states do not fit even with a micro-batch of one, accumulation will not rescue the run.

A concrete example

Suppose a language-model fine-tuning job processes sequences of 2,048 tokens. Profiling shows that one GPU can fit four sequences, but a target batch of 32 sequences gives better training behaviour.

Set the micro-batch size to 4 and accumulate for 8 steps:

4 sequences per micro-batch x 8 micro-batches = 32 sequences per optimizer update

A simple PyTorch loop looks like this:

import torch

accumulation_steps = 8
optimizer.zero_grad(set_to_none=True)

for step, batch in enumerate(dataloader):
    logits = model(batch["input"])
    loss = criterion(logits, batch["label"])  # mean loss for this micro-batch

    (loss / accumulation_steps).backward()

    is_update = (step + 1) % accumulation_steps == 0
    if is_update:
        torch.nn.utils.clip_grad_norm_(
            model.parameters(),
            max_norm=1.0,
        )
        optimizer.step()
        optimizer.zero_grad(set_to_none=True)

The call to backward() adds to existing .grad buffers. It does not overwrite them. The call to zero_grad() is deliberately placed only after the optimizer update.

The code assumes that the number of micro-batches is divisible by 8. If the data loader ends with, say, three leftover micro-batches, you must either discard that incomplete window deliberately or divide those three losses by 3 and perform a final update. Otherwise, the final gradients are scaled down by 8, or worse, they may never be applied.

Also log the original loss, not only the divided loss. If the real micro-batch loss is 2.4, the value used for backpropagation in an eight-step window is 0.3. Reporting 0.3 as the training loss makes the dashboard look impressively calm and completely misleading.

With mixed-precision training using GradScaler, divide before scaling. Unscale before gradient clipping:

scaler.scale(loss / accumulation_steps).backward()

if is_update:
    scaler.unscale_(optimizer)
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    scaler.step(optimizer)
    scaler.update()
    optimizer.zero_grad(set_to_none=True)

The scaler is updated at the optimizer boundary, not after every micro-batch.

Effective batch size

For data-parallel training, the usual formula is:

effective batch size = per-device micro-batch size x accumulation steps x number of devices

In the example, one GPU gives:

4 x 8 x 1 = 32 sequences

If four distributed-training processes each handle four sequences, the global effective batch is:

4 x 8 x 4 = 128 sequences

Standard distributed data parallel training already averages gradients across processes. Do not divide by the number of devices a second time unless your custom communication code specifically requires it.

Hugging Face Trainer expresses the same pattern directly:

from transformers import TrainingArguments

args = TrainingArguments(
    output_dir="runs/gradient-accumulation",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
)

On one device, that is an effective batch of 32. On four devices, it is 128.

For long-context language models, examples are not always equally expensive. One sequence may contain 2,048 valid tokens while another contains 700 tokens and padding. Averaging each micro-batch loss equally is not always the same as averaging every valid token equally. For exact token-level normalisation, accumulate the unreduced loss sum and divide by the total number of non-padding tokens in the effective batch. This is a common source of small but persistent differences between custom loops and framework trainers.

The senior-level nuance

Gradient accumulation can match a true larger batch only under useful conditions: the micro-batches have the same size, the loss is correctly normalised, the model weights do not change between them, and the model has no behaviour that depends on the physical batch.

Batch Normalization is the classic exception. BatchNorm computes a mean and variance from the examples in the current physical batch. Eight micro-batches of four examples calculate eight separate sets of statistics. One true batch of 32 calculates one set. The gradients are therefore not equivalent, even though the accumulated gradient buffers have the same shape. LayerNorm, commonly used in Transformer models, does not have this particular batch-size dependency.

Randomness also means the runs will not be bit-for-bit identical. Dropout masks and floating-point addition order can differ. Usually that is ordinary training noise, not a conceptual failure.

Gradient clipping must happen after accumulation if you want to clip the effective gradient. Clipping each micro-batch separately changes the result because:

clip(g1) + clip(g2) is generally not the same as clip(g1 + g2)

Learning-rate schedulers need the same treatment. If you call scheduler.step() after every micro-batch while accumulating for eight steps, the schedule advances eight times faster than the optimizer. Warmup can finish prematurely, and a cosine schedule can reach its minimum long before training is complete. Count optimizer updates, not data-loader iterations, when designing the schedule.

The same principle applies to logging, checkpoint cadence, and some forms of weight decay. Accumulation reduces the number of optimizer updates per epoch. Adam’s momentum estimates and AdamW’s decoupled weight decay are updated on those optimizer steps, not on each individual forward pass.

There is also a throughput trade-off. A real batch lets the GPU process more examples in parallel. Accumulation processes the same examples in several sequential passes, adding kernel-launch overhead and leaving some hardware underused. If the desired physical batch already fits, using it directly is often faster. Distributed data parallelism, activation checkpointing, or parameter sharding may be better solutions when throughput or model size is the real constraint.

Failure modes to recognise

The most common bug is forgetting to divide the loss. The immediate symptom may be an apparently healthy falling training loss. After increasing accumulation from 1 to 8, however, updates become much larger: the loss starts oscillating, gradients hit clipping constantly, or the run produces NaNs.

A second bug is stepping the scheduler or clipping gradients inside the micro-batch loop. The model may train, but the learning-rate curve finishes too early or the final accuracy changes sharply when accumulation is changed.

Accumulation should not make memory grow after every micro-batch. If GPU memory rises steadily until an out-of-memory error, look for retain_graph=True, tensors with attached computation graphs stored in a Python list, or logging code that saves loss instead of loss.item(). The whole point is to release each micro-batch’s activations after its backward pass.

In multi-GPU training, synchronising gradients after every micro-batch can also destroy throughput. Distributed frameworks often provide a way to suppress communication for the first micro-batches and synchronise only at the boundary. The exact setting depends on the framework, so verify its current distributed-training documentation.

What they’ll ask next

Is gradient accumulation exactly the same as training with a larger batch?

Not always. It is close when micro-batches are equally sized, the loss is linear and correctly normalised, and the model has no batch-dependent layers. BatchNorm, variable token counts, clipping placement, dropout randomness, and floating-point order can make the result different.

A job uses a micro-batch of 6, four accumulation steps, and three GPUs. What is the effective batch size?

6 x 4 x 3 = 72 examples per optimizer update. That is the global batch, assuming every process receives six examples and standard distributed data parallel gradient averaging is used.

Where should gradient clipping and the learning-rate scheduler be called?

After the accumulated gradient is complete and immediately around the optimizer update. Unscale mixed-precision gradients before clipping, call optimizer.step(), then advance the scheduler once.

When would you not use accumulation?

When the target batch already fits and direct batching is faster, when the task needs a weight update after every example, or when the model’s batch-dependent operations make accumulated and physical batches behave too differently. It also cannot solve a model that fails to fit with even one micro-batch.

Say this in the interview: Gradient accumulation lets several memory-sized micro-batches share one optimizer update, giving a larger effective batch without storing all its activations, but the loss, scheduler, clipping, and batch-dependent layers must be handled carefully.