datarekha
Deep Learning Hard Asked at PyTorchAsked at NVIDIAAsked at Meta

What correctness issues should you consider when using activation checkpointing?

For research-engineerML Engineer
The short answer

A checkpointed region is replayed during backward, so it should behave consistently from its inputs. Side effects, changing global state, divergent control flow, unexpected device movement, and random operations can break equivalence. PyTorch preserves RNG state for operations such as dropout by default; choose reentrant behavior explicitly and test gradients after changing boundaries.

How to think about it

A checkpointed region may execute once in the original forward pass and again during backward. It must not silently produce a different function. Avoid externally visible side effects and changing global state. Be careful with data-dependent branches and tensors moved to devices that the checkpoint machinery did not anticipate.

Random operations matter: PyTorch normally saves and restores RNG state so dropout replay matches the original forward. Disabling preservation can improve performance but changes equivalence. Current PyTorch APIs also require an intentional use_reentrant choice.

Learn it properly Activation checkpointing

Keep practising

How does gradient checkpointing reduce GPU memory, and what is the trade-off? What is the difference between activation checkpointing and gradient accumulation? How does dropout work, and why must it behave differently during training and inference? How do you achieve reproducibility in ML training pipelines — covering seeds, environment, and data versioning? In a PyTorch training loop, why do you need to call optimizer.zero_grad() before backpropagation?
All Deep Learning questions

Explore further

Retraining & continual learning Persistence & streaming Autograd
Dropout Vanishing Gradient Exploding Gradient Gradient Clipping
Skip to content