What is sequence padding and why is it necessary for batch training?

How to think about it

Short answer

Sequence padding adds a special non-content token, usually called [PAD], to shorter token sequences so every example in a batch has the same length. It is necessary for the usual dense-tensor training path because operations such as matrix multiplication need rectangular arrays, but padding is not universally necessary: packed and ragged representations can handle variable lengths directly.

Why batching creates this problem

A token is a small unit of text, such as a word or word fragment, represented inside the model by an integer ID. Suppose tokenization produces these two examples:

• Example A: five tokens
• Example B: three tokens

Without padding, the batch is a list containing one tensor with shape [5] and another with shape [3]. There is no single second dimension for the framework to use. A dense batch needs one shape, such as [batch_size, sequence_length].

Padding changes the second example to five positions:

Example A: [17, 42, 9, 6, 3]
Example B: [17, 81, 4, 0, 0]

Here, 0 represents [PAD]. The input tensor now has shape [2, 5]: two examples and five positions each. After an embedding layer with hidden width 768, the representation has shape [2, 5, 768].

That regular shape is what lets a GPU process the two examples together. The model can perform one batched embedding lookup, one batched linear operation, and one batched attention operation instead of running a separate computation for every length.

The important qualification is that neural networks are not fundamentally incapable of variable-length input. The constraint comes from the common implementation: dense tensors and highly optimized batched kernels. PyTorch packed sequences, nested or ragged tensors, and specialized variable-length attention kernels can reduce or avoid padding. They also make batching, masking, compilation, and distributed training more complicated.

A concrete batch

The following example builds a padded batch without relying on a tokenizer, so the shapes are completely checkable:

import torch

input_ids = torch.tensor([
    [17, 42, 9, 6, 3],
    [17, 81, 4, 0, 0],
])

attention_mask = (input_ids != 0).long()

print(input_ids.shape)
print(attention_mask)

The output is:

torch.Size([2, 5])
tensor([[1, 1, 1, 1, 1],
        [1, 1, 1, 0, 0]])

The first row contains five real tokens. The second contains three real tokens and two padding positions. The attention_mask records that difference: 1 means a real token and 0 means padding.

With a Hugging Face tokenizer, the equivalent operation is commonly expressed with padding=True or padding="longest". The tokenizer returns both input_ids and an attention mask when requested. The exact padding token ID depends on the tokenizer.

Padding does not mean “ignore”

Adding [PAD] creates the rectangular shape. It does not, by itself, tell the model that the new positions are meaningless.

A self-attention layer compares every query position with every key position. If the second example has two padding positions and no mask is supplied, a real word can assign attention weight to those positions. The model then consumes the embedding for [PAD] as though it were useful input.

An attention mask prevents this. Internally, implementations usually apply the padding mask to the attention scores before softmax. Scores for padded key positions are made effectively impossible to select, so their attention weights become approximately zero.

There is a subtle detail here: the mask mainly prevents real tokens from attending to padded keys. The model may still compute output vectors at padded query positions. That is fine if those vectors are ignored when pooling and when calculating loss. Padding is not physically removed from the tensor; it is excluded from the operations where it could matter.

Common trap: an attention_mask is not automatically a loss mask.

For language modeling or token classification, the model often produces one loss value per position. In the example above, there are ten total positions but only eight real tokens. If the two padded positions are included in the average, the reported loss contains meaningless terms. The model can also receive gradients for learning to predict [PAD].

A standard solution is to put an ignored label, commonly -100, at padded label positions and pass the same value as ignore_index to cross-entropy loss. The loss then averages only over real targets. In a custom loss, multiply by the token mask and divide by the number of real tokens, not by the total padded length.

For sequence classification, there is usually one label per example rather than one label per token, so there are no padded labels to ignore. The model still needs correct masking when it pools token representations. Otherwise, the classifier may use the number or contents of padding positions as accidental evidence.

Decoder-only language models have another mask as well: a causal mask, which prevents a token from looking at future tokens. Causal masking and padding masking solve different problems. A correct decoder often uses both.

Pre-padding versus post-padding

Post-padding places [PAD] after the real sequence:

[17, 81, 4, 0, 0]

Pre-padding places it before the real sequence:

[0, 0, 17, 81, 4]

For encoder-style Transformers, post-padding is common because it is simple and works naturally with the usual position numbering. With a correct attention mask, either side can work, but positional embeddings and model-specific conventions still matter.

For recurrent neural networks, padding direction interacts with how the final hidden state is chosen. If code blindly takes the hidden state at the final array position, post-padding means that final position is a pad, not the last real token. Packed sequences or explicit sequence lengths solve this properly. Switching to pre-padding is not a universal fix, because the recurrent network would then process padding before the real tokens.

Decoder-only generation often uses left, or pre-, padding. The reason is practical: when generating the next token, the last real token should line up at the same right-hand position across the batch. With right-padding, a shorter example ends in [PAD], so naïve code that selects the final column may try to generate from padding. Left-padding avoids that alignment problem, but position IDs and the attention mask must also be handled correctly.

So the interview-quality answer is not “post-padding is always better” or “pre-padding is for reverse reading.” The correct choice depends on the architecture, the position-encoding scheme, and how the model selects its final or next-token representation.

Static, dynamic, and bucketed padding

Static padding uses one global length, such as 512, for every example. It gives stable tensor shapes and can simplify compilation, caching, and distributed execution. Its cost is wasted computation.

Consider a batch of 32 examples whose useful length is about 64 tokens. Padding every example to 512 creates:

• 32 × 64 = 2,048 useful-length slots
• 32 × 512 = 16,384 padded slots

That is eight times as many token positions. For the self-attention score matrix, whose work grows roughly with the square of sequence length, the comparison is approximately 512² / 64² = 64 times as many score positions. The total model cost is not exactly 64 times higher because feed-forward layers scale linearly, but the waste can still be severe.

Dynamic padding pads each batch only to the longest sequence in that batch. If the batch contains lengths 42 and 10, the framework uses length 42 rather than a global maximum. This usually saves memory and computation when sequence lengths vary widely.

Dynamic padding is not guaranteed to be faster. If almost every sequence is already close to the maximum, there is little to save. Highly dynamic shapes can also cause extra compilation or reduce the benefit of specialized kernels. Some production systems therefore combine dynamic padding with length bucketing: examples with similar lengths are grouped together, while batches remain reasonably well shuffled.

Some hardware and kernels prefer lengths aligned to particular multiples. Padding to a multiple of 8, 16, or another hardware-friendly size can improve throughput in some setups, but it adds waste and should be measured rather than assumed.

Truncation is the opposite operation. Padding makes short sequences longer; truncation makes long sequences shorter so they fit a limit. Truncation can discard important context, so a production system may use a sliding window, a task-specific truncation rule, or a longer-context model instead of silently dropping the beginning or end.

Failure modes to recognize

First symptom	Likely cause	Practical fix
The data loader fails while stacking examples of different lengths	No padding or packing strategy was supplied	Pad in the collator or use a variable-length representation
The same sentence behaves differently depending on which longer sentence shares its batch	The padding mask is missing, inverted, or has the wrong shape	Inspect input_ids and verify that real tokens are 1 and pads are 0
Training loss falls suspiciously quickly and token accuracy looks excellent	Padded label positions are included in the loss	Set padded labels to the ignored value and verify the loss denominator
A decoder tokenizer raises an error when asked to pad	It has no configured padding token	Configure padding deliberately, often with a suitable existing token, and mask those positions in the loss

A useful debugging check is to print one complete batch: input_ids, attention_mask, and labels. Count the 1 values in the mask. That count should equal the number of real tokens. Then confirm that the corresponding label positions are the only ones contributing to loss.

What they’ll ask next

Is padding required for every model?
No. It is required by the standard dense rectangular batching path. Packed sequences, ragged tensors, and specialized variable-length kernels can avoid it, but they add implementation complexity and are not equally supported across models and hardware.

Does the attention mask also prevent padding from affecting the loss?
Usually no. The attention mask controls which positions participate in attention. Token-level loss needs its own ignored labels or an explicit loss mask.

Should I always use dynamic padding?
No. Use it when sequence lengths vary enough to create waste. Static padding can be reasonable when lengths are uniform or when stable shapes materially simplify compilation and serving. Bucketing often gives a useful middle ground.

Say this in the interview

“Padding makes variable-length examples rectangular so dense batch operations can run together; the attention mask excludes padded positions from attention, and a separate loss mask excludes them from gradients, while packed or ragged representations can avoid padding altogether.”