Walk me through the forward pass of a neural network end-to-end.

How to think about it

At a high level, a forward pass takes input features, applies each layer’s learned parameters and nonlinear operations in order, and produces an output. During training, that output is compared with the label to form a loss; the framework records the computation and needed intermediate values so the backward pass can later calculate gradients.

The mechanism

For a typical dense layer, I would describe the computation as an affine transformation followed by an activation function. An affine transformation means a matrix multiplication plus a bias. An activation function adds a nonlinearity.

For layer l:

z^[l] = W^[l] · a^[l-1] + b^[l]
a^[l] = f^[l](z^[l])

Here, a^[0] is the input x. W contains the learned weights, b is the learned bias, z is the pre-activation value, and a is the layer output after applying f. The bracketed superscript identifies the layer; it is not an exponent.

The weights determine how strongly each input matters. The bias shifts the result before the activation. ReLU, for example, returns the value when it is positive and zero otherwise. That simple gate is important: if every layer only performed an affine transformation, several layers would collapse mathematically into one affine transformation. The network could not represent curved or otherwise nonlinear decision boundaries.

Shapes are part of the forward pass, not bookkeeping added afterward. Suppose a batch contains B examples, each with d features, and the layer has h neurons:

• input: [B, d]
• weights: [h, d]
• bias: [h]
• output: [B, h]

In column-vector notation the equation uses W · a. In PyTorch’s usual batch-oriented notation, the same operation appears as x @ weight.T + bias. The transpose is why the handwritten formula and framework code can look reversed.

A concrete forward pass

Consider a toy fraud classifier. Each transaction has two already-normalized features: an amount score of 2 and a device-risk score of 1. The hidden layer has two neurons, and the output layer predicts two classes: legitimate, class 0, or fraudulent, class 1.

import torch
import torch.nn.functional as F

x = torch.tensor([[2.0, 1.0]])   # one example, shape [1, 2]
y = torch.tensor([1])            # the correct class is 1

W1 = torch.tensor([[1.0, -1.0],
                   [0.5,  2.0]])
b1 = torch.tensor([0.0, -1.0])

W2 = torch.tensor([[ 1.0, -0.5],
                   [-1.0,  0.5]])
b2 = torch.tensor([0.0, 0.5])

z1 = x @ W1.T + b1
a1 = torch.relu(z1)
logits = a1 @ W2.T + b2
loss = F.cross_entropy(logits, y)

The first affine operation gives:

z1 = [1, 2]

The ReLU activation leaves both values positive:

a1 = [1, 2]

The second affine operation produces:

logits = [0, 0.5]

A logit is an unnormalized score for a class. It is not yet a probability and not yet a final class decision. The larger logit belongs to class 1, so an argmax decision would classify this transaction as fraudulent.

For intuition, applying softmax would turn the logits into probabilities of about 0.3775 for class 0 and 0.6225 for class 1. Because the correct label is class 1, the cross-entropy loss is approximately:

-log(0.6225) = 0.4741

That is the end of the model’s forward computation for this example. The loss is calculated immediately afterward so the model can be trained, but strictly speaking the model’s forward method normally returns the logits.

The output depends on the task

The final layer is not always followed by the same activation. The output and loss must match the prediction problem.

Task	Model output	Common PyTorch loss
Multiclass classification	One logit per class	CrossEntropyLoss
Binary classification	One logit	BCEWithLogitsLoss
Regression	One or more real-valued outputs	MSELoss or L1Loss

For multiclass classification, CrossEntropyLoss expects raw logits. It internally performs the numerically stable log-softmax operation and then computes the negative log-likelihood. That is why the example does not call softmax before F.cross_entropy.

For a regression model, adding softmax would make no sense because the output is a number such as a house price or temperature. For a binary classifier, a sigmoid probability may be useful for display, but BCEWithLogitsLoss should receive the raw logit because it combines sigmoid and the loss in a more stable calculation.

A typical PyTorch training step therefore looks like this:

model.train()

logits = model(x_batch)
loss = F.cross_entropy(logits, y_batch)

optimizer.zero_grad()
loss.backward()
optimizer.step()

The forward pass produces logits. The loss measures their error. backward() computes gradients, and step() changes the weights. The forward pass itself does not update any parameter.

Why the framework stores intermediate values

When PyTorch runs the forward pass with gradient tracking enabled, autograd builds a computation graph. Each operation knows which earlier values it depends on and how to differentiate itself.

For a dense layer, the backward pass needs the previous activation. In simplified form:

dW = dz · a_prev.T
db = dz
da_prev = W.T · dz

dz is the gradient arriving at the layer’s pre-activation. The previous activation is needed to calculate the weight gradient. ReLU also needs to know which pre-activation values were positive, because gradients pass through those positions and are blocked at the others.

The framework does not necessarily retain every tensor indiscriminately. It saves the tensors required by the recorded operations. This is why a forward pass can consume substantial GPU memory even though it appears to be “only making predictions.”

Gradients are accumulated in parameter tensors during loss.backward(). The optimizer uses those gradients afterward. This distinction matters when debugging: if the loss changes but the weights never change, the problem may be in the optimizer step rather than the forward pass. For a deeper treatment of the graph and gradients, see autograd.

Training mode, evaluation mode, and inference memory

The forward computation can behave differently in training and evaluation.

model.train() enables training behavior such as dropout and batch statistics in batch normalization. Dropout randomly removes some activations during training to reduce reliance on any one path through the network. model.eval() disables dropout and tells batch normalization to use its stored running statistics.

model.eval() does not disable gradient tracking. For ordinary inference, use both:

model.eval()

with torch.no_grad():
    logits = model(x_batch)

Without no_grad, PyTorch may construct a graph that is unnecessary for prediction. If an application stores those output tensors, it can also keep their graphs alive and make GPU memory grow across requests. If the application needs gradients for attribution or adversarial analysis, then disabling gradients is intentional and should not be done.

A common production failure is forgetting model.eval(). The first symptom is often that repeated calls on the same input produce different outputs, or that validation accuracy is noticeably lower than expected. Dropout is still active, or batch normalization is using statistics from the current batch rather than its learned running values.

The senior-level nuance

The dense-layer equation is the common interview explanation, but modern networks are not always a simple chain of matrix multiplications. A convolution applies local weighted operations over an image. An attention block creates queries, keys, and values, then mixes information across positions. A residual block adds a skip path, so its output may look like f(x) + x.

The general idea remains the same: the forward pass evaluates a directed computation graph from inputs and parameters to outputs. A graph can branch and merge; it does not have to be a straight line.

There is also a memory-versus-compute trade-off. Training saves intermediate activations so backward can use them. Activation checkpointing saves fewer of them and recomputes some during the backward pass. That lowers memory usage but increases computation and can increase training time. It is useful when a large model or long sequence does not fit in GPU memory.

What they’ll ask next

Why do you return logits instead of probabilities from a classifier?

Because the loss function can apply the appropriate normalization in a numerically stable way. In PyTorch, CrossEntropyLoss expects raw multiclass logits, while BCEWithLogitsLoss expects a raw binary logit. I apply softmax or sigmoid only when I need probabilities for reporting or downstream logic.

Is the loss part of the forward pass?

The model’s forward method usually ends when it returns its output, such as logits. In a training step, I immediately pass that output and the label to a loss function. People often call the combined model-output-and-loss calculation the training forward pass, but backpropagation and the optimizer update are separate steps.

What exactly does autograd cache?

It records the operations in the computation graph and saves the intermediate tensors needed to differentiate them. For a dense layer, that commonly includes the input activation and enough information about the activation function. Those saved values let backward calculate weight, bias, and input gradients without rerunning the entire forward pass.

Say this in the interview

“A forward pass carries the input through each layer’s affine operation and activation, produces task-specific outputs such as logits, and usually feeds them into a loss during training; autograd records the needed operations and activations so backward can compute gradients, while inference uses evaluation mode without gradient tracking.”