What is the difference between activation checkpointing and gradient accumulation?
Activation checkpointing reduces saved activations inside one micro-batch by recomputing forward regions during backward. Gradient accumulation reduces the micro-batch processed at once and sums gradients across several micro-steps before one optimizer update, preserving a larger effective batch. They target different memory pressure and can be combined.
How to think about it
Activation checkpointing reduces saved tensors within one forward/backward pass by recomputing selected regions. Gradient accumulation runs several smaller forward/backward micro-steps, retains accumulated gradients, and delays optimizer.step() until the desired effective batch has been covered.
They compose: use a smaller micro-batch with accumulation, then checkpoint the remaining per-micro-batch activations. Neither directly shards parameters or optimizer state.