Autograd — why PyTorch exists

Your model has 175 million parameters. The loss is one number. After the forward pass, you need to know how that one number should change with respect to all 175 million parameters.

Writing those derivatives by hand is possible. It is also how you end up spending Tuesday fixing a sign error in layer 37.

Before automatic differentiation, every new layer needed a forward formula and a matching backward formula. PyTorch’s central promise is simpler:

For differentiable PyTorch operations executed with gradient tracking enabled, you write the forward computation; PyTorch records the operation history and applies the registered backward rules.

That system is autograd, short for automatic differentiation.

The mental model

Autograd is not symbolic algebra or numerical guessing. It is the chain rule applied to the operations that actually ran.

import torch

x = torch.tensor(3.0, requires_grad=True)
y = x ** 2 + 2 * x + 1

y.backward()

print(y)      # tensor(16., grad_fn=<AddBackward0>)
print(x.grad) # tensor(8.)

requires_grad=True asks PyTorch to track operations downstream of x so it can later calculate derivatives involving x.

During the forward pass, PyTorch records that y came from a square, a multiplication, and additions. Each operation has a local derivative rule. When y.backward() runs, those rules are applied in reverse.

A leaf tensor is an input created by you rather than produced by another tracked operation. Model parameters are usually leaves. A non-leaf tensor is an intermediate result, such as x ** 2 or a layer activation.

After backward, PyTorch normally stores gradients in .grad on leaf tensors. Intermediate tensors participate in the calculation, but their .grad is not retained unless you call .retain_grad(). Thus, a missing .grad on an activation is often normal; a missing .grad on a trainable parameter deserves investigation.

One example, all the way through

At x = 3, the expression has these intermediate values:

• u = x² = 9
• v = 2x = 6
• s = u + v = 15
• y = s + 1 = 16

Work backward through both branches:

• Through u = x²: dy/dx = 1 × 1 × 2x = 6
• Through v = 2x: dy/dx = 1 × 1 × 2 = 2

Both paths end at the same x, so their contributions are added:

dy/dx = 6 + 2 = 8

The graph is a diamond: x is used twice, so its gradient receives one contribution from each use. Autograd must accumulate both.

The graph is built during the forward pass, not before it. Ordinary Python control flow therefore works naturally:

if temperature > 0.5:
    output = hot_path(x)
else:
    output = cold_path(x)

Only the operations on the path that ran are recorded for that pass. The next forward pass can build a different graph.

Why reverse mode wins for neural networks

A neural network has millions of parameters but usually produces one loss value. You need one derivative of that loss with respect to every parameter.

Forward-mode differentiation carries a separate derivative for each input direction. That is useful with few inputs and many outputs. Reverse mode starts with the one loss and sends one gradient signal backward, multiplying it by each operation’s local derivative.

For tensor operations, this is a vector-Jacobian product (VJP): an upstream gradient vector multiplied by the operation’s Jacobian. The Jacobian lists every output derivative with respect to every input, but PyTorch computes the needed product without constructing the whole matrix. For a scalar loss, the initial upstream gradient is 1.

A vector output has no single default derivative. Supply an upstream vector, or reduce it to a scalar with .mean() or .sum().

Build a tiny autograd

Remove tensors and keep only numbers. Each Node stores its value, parents, local derivatives, and an accumulated gradient:

class Node:
    def __init__(self, value, parents=()):
        self.value = float(value)
        self.parents = parents
        self.grad = 0.0

    def __add__(self, other):
        other = other if isinstance(other, Node) else Node(other)
        return Node(
            self.value + other.value,
            parents=[(self, 1.0), (other, 1.0)],
        )

    def __mul__(self, other):
        other = other if isinstance(other, Node) else Node(other)
        return Node(
            self.value * other.value,
            parents=[(self, other.value), (other, self.value)],
        )

    def __pow__(self, n):
        return Node(
            self.value ** n,
            parents=[(self, n * self.value ** (n - 1))],
        )

    def backward(self, upstream=1.0):
        self.grad += upstream
        for parent, local in self.parents:
            parent.backward(upstream * local)


x = Node(3.0)
y = x ** 2 + x * 2 + Node(1.0)

y.backward()

print(f"y = {y.value}")
print(f"dy/dx = {x.grad}")

y = 16.0
dy/dx = 8.0

The important line is:

parent.backward(upstream * local)

The child passes its incoming gradient multiplied by its local derivative. That is the chain rule in executable form. Because x appears in both branches, its backward() runs twice, and self.grad += upstream adds the contributions.

This toy implementation omits tensors, broadcasting, device kernels, memory management, topological scheduling, saved values, in-place checks, and thousands of derivative definitions. It still captures the mechanism: PyTorch attaches backward metadata to operation results, saves values needed by backward rules, and executes those rules efficiently over tensor data.

The production training pattern

A training step has two separate jobs:

• backward() computes gradients and places them in parameter .grad fields.
• optimizer.step() reads those gradients and changes the parameters.

for xb, yb in train_loader:
    optimizer.zero_grad(set_to_none=True)

    prediction = model(xb)
    loss = criterion(prediction, yb)

    loss.backward()
    optimizer.step()

Gradients accumulate by default. If one batch produces 0.3 and the next -0.1, the parameter holds 0.2 unless you clear it. This enables intentional gradient accumulation over several micro-batches, but ordinary training loops need zero_grad() before the next update.

The graph is separate from the gradients. After normal backward, PyTorch usually frees the graph’s saved intermediates while leaving parameter .grad values. Each batch therefore needs a fresh forward pass and a fresh graph.

When you do not want gradients

model.eval() and torch.no_grad() solve different problems.

model.eval() changes module behavior: dropout stops masking activations, and batch normalization uses stored running statistics. It does not disable autograd.

torch.no_grad() prevents ordinary operations inside its block from being recorded for backward:

model.eval()

with torch.no_grad():
    pred = model(x)

This saves memory and graph-construction work. It does not change whether the model is in training mode. For serving or a pure validation pass, torch.inference_mode() also disables autograd and removes additional bookkeeping for tensors that will not later re-enter gradient calculations. no_grad() is more flexible when ordinary tensor behavior may still be needed.

Neither context changes weights or performs an optimizer step.

Storing predictions without disabling gradients can keep every graph alive:

predictions = []

for xb, _ in validation_loader:
    predictions.append(model(xb))

Use no_grad(), or store model(xb).detach() when appropriate.

detach() cuts one path

detach() keeps a tensor’s numerical value but treats it as constant for future gradient calculations. This is useful when two models interact but only one should learn:

teacher_logits = teacher(x).detach()
student_logits = student(x)

loss = loss_function(student_logits, teacher_logits)
loss.backward()

The loss depends numerically on both tensors, but the detached teacher tensor has no connection back to the teacher’s graph. The student receives gradients; the teacher does not.

If the teacher is only producing targets, prevent its graph from being built:

with torch.no_grad():
    teacher_logits = teacher(x)

The difference is timing:

• no_grad() prevents operations in a region from being recorded.
• detach() severs one tensor’s history after it has been produced.

A detached tensor usually shares storage with its source. For independent data, use tensor.detach().clone(). loss.item() returns a detached Python scalar, so it is safe for logging if you keep the original loss tensor. Do not replace loss with loss.item() or call backward() on the returned number.

Failures you can diagnose from the first symptom

“This tensor does not require grad”

This error means the loss is disconnected from every trainable value. Common causes include torch.no_grad(), .detach(), conversion through NumPy, .item(), or freezing all parameters.

print(loss.requires_grad)
print(loss.grad_fn)

For a normal training loss, requires_grad should be true and grad_fn should describe its producing operation. Setting requires_grad on the final number cannot recreate history that was already cut.

“Trying to backward through the graph a second time”

Normal backward frees saved intermediate values. Calling backward again on the same result often fails. Run the forward pass again to create a fresh loss. retain_graph=True is for legitimate multiple backward passes, but it keeps activations in memory and is not a general repair.

.grad is None

On a parameter, check whether it was frozen, used in this branch, or disconnected from the loss. On an intermediate activation, None is normal unless you request retention:

activation.retain_grad()

PyTorch avoids storing intermediate gradients by default to reduce memory use.

Gradients are much larger than expected

First check optimizer.zero_grad(): accumulated gradients grow with the number of batches since the last reset. If accumulation is intentional, scale the loss consistently.

Also check loss reduction. A per-example loss summed over a batch grows with batch size; a mean-reduced loss generally does not.

“Modified by an inplace operation”

Some backward rules need a saved forward value. Mutating that value in place before backward makes it unreliable. Prefer out-of-place operations while debugging, and remember that a detached view may share storage with its source.

The honest boundary

Autograd differentiates the computation you wrote. It cannot tell whether that computation represents the right objective. You can receive valid gradients for wrong labels, leakage, a sign error, or a metric used accidentally as the loss. Successful backward means the derivative machinery worked, not that the model is learning the intended thing.

Integer tensors do not carry ordinary derivatives. Some operations are nondifferentiable at particular points, where PyTorch uses a defined convention or subgradient. Converting to NumPy and back breaks the graph unless you provide a differentiable custom operation.

Autograd makes derivatives available, not free to store. Training retains activations for backward, so it uses more memory than inference. Mixed precision and activation checkpointing can change that trade-off, but they are separate engineering choices.

Quick check