How does gradient checkpointing reduce GPU memory, and what is the trade-off?
Gradient checkpointing—more precisely activation checkpointing—keeps selected forward boundaries instead of every intermediate activation. During backward it re-runs checkpointed forward regions to reconstruct discarded tensors. It reduces activation memory but leaves parameters, gradients and optimizer state unchanged, trading extra computation for lower peak VRAM.
How to think about it
Gradient checkpointing—more precisely activation checkpointing—keeps selected forward boundaries instead of every intermediate activation. During backward it re-runs checkpointed forward regions to reconstruct discarded tensors. It reduces activation memory but leaves parameters, gradients and optimizer state unchanged, trading extra computation for lower peak VRAM.
The amount saved depends on checkpoint placement, sequence length, micro-batch size and architecture. The slowdown is not a fixed constant; benchmark peak memory and tokens per second on representative shapes.