Why does a transformer need positional encoding?

How to think about it

Because self-attention has no built-in clock: it compares token content and mixes value vectors, but it does not know whether a token came first or seventeenth. A positional signal supplies that missing coordinate, allowing a transformer to represent order-dependent facts such as whether Alice paid Bob or Bob paid Alice.

Why self-attention does not know order

Each input token begins as an embedding, a vector representing its content. In one attention head, that vector is projected into a query, what the token is looking for; a key, how the token can be matched; and a value, the information passed to other tokens.

The head computes:

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k))V

Here, d_k is the key-vector width, and softmax converts each row of similarity scores into weights that sum to one.

Notice what is absent: the index of each token. The score between two tokens depends on their query and key vectors, not on whether they occupy positions 2 and 3 or positions 20 and 21.

Suppose P is a permutation matrix, meaning a matrix that reorders rows. If the input rows are reordered, the queries, keys, and values are reordered too. The attention score matrix is reordered in the corresponding rows and columns, and the output is reordered in the same way:

f(PX) = P f(X)

This property is called permutation equivariance. The output still has one row for each input token, but the computation has no way to treat a row differently merely because it is the first row or the seventh row.

That is different from an RNN. At step t, an RNN receives the hidden state from step t - 1, so sequence order is built into its recurrence. A transformer processes all tokens in parallel. Parallelism is the point, but it means order must be supplied separately.

Adding more transformer layers does not repair this. Residual connections, layer normalization, feed-forward layers, and multi-head attention all apply the same operation to every position. Stacking more position-blind layers preserves the same symmetry.

Common misconception. Saying that attention sees a “bag of tokens” is shorthand. The rows do not literally disappear or get randomly rearranged. Their order remains in the tensor layout. What is missing is a usable feature saying “this token is at position 2,” so shared transformer blocks cannot use slot identity as part of their reasoning.

A concrete numerical example

Take a deliberately tiny attention head. Give the three tokens in Alice paid Bob one-hot embeddings:

Alice = [1, 0, 0]
paid  = [0, 1, 0]
Bob   = [0, 0, 1]

Q = K = V = X
d_k = 3

A token matches itself with a dot product of 1, and matches either other token with a dot product of 0. After scaling by sqrt(3), the matching score is about 0.577, while the other two scores are 0.

Softmax therefore gives each row approximately these weights:

matching token       0.471
each other token     0.264

For Alice paid Bob, the output rows are approximately:

Alice row = [0.471, 0.264, 0.264]
paid row  = [0.264, 0.471, 0.264]
Bob row   = [0.264, 0.264, 0.471]

Now reverse the names: Bob paid Alice. The rows become:

Bob row   = [0.264, 0.264, 0.471]
paid row  = [0.264, 0.471, 0.264]
Alice row = [0.471, 0.264, 0.264]

The rows have changed places, but the token-associated computations have not changed. If a classifier averages the three output rows, both sentences produce the same average: each coordinate is about 0.333.

That is the core problem. The two sentences contain the same tokens but express different transactions. A position-agnostic pooling operation cannot tell who paid whom.

A positional signal changes the input to:

z_i = e_i + p_i

e_i is the token embedding and p_i is the positional vector for slot i. For the first sentence, Alice might produce e_Alice + p_1; in the reversed sentence, Alice produces e_Alice + p_3. The same word at a different location now creates different queries and keys. Attention can learn patterns such as “the subject usually appears before the verb” or “look two positions to the left.”

Common ways to provide position

The original Transformer added fixed sinusoidal vectors to the token embeddings:

PE(pos, 2i)   = sin(pos / 10000^(2i / d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i / d_model))

pos is the token’s position, i identifies a pair of embedding channels, and d_model is the model width. Each pair uses a different wavelength. Together, the sine and cosine values give each position a distinctive pattern that changes smoothly with distance.

A useful mathematical property is that the dot product between the encodings for positions p and q depends on their offset, p - q, rather than on the two absolute numbers independently. That makes relative relationships geometrically consistent. Because the formula can be evaluated for a new position, it can produce vectors beyond the length seen during training. That does not guarantee good extrapolation; the model may still have learned behaviors tied to its training context.

Other approaches are common:

Scheme	How position enters	Main strength	Main caveat
Learned absolute	Add a learned vector for each slot	Flexible and simple	The table has a natural maximum length
Sinusoidal	Add fixed sine and cosine vectors	No learned lookup table	Longer-context performance is not guaranteed
RoPE	Rotate queries and keys by position-dependent angles	Attention scores carry relative position	Context extension often needs careful scaling
ALiBi	Add a distance-based bias to attention logits	Simple relative distance preference	It is not a full absolute position representation

BERT and early GPT-style systems commonly used learned absolute position embeddings. Many modern decoder language models use RoPE, short for Rotary Position Embedding. RoPE does not add a position vector directly to the token embedding. Instead, it rotates query and key coordinates by an angle determined by position. When the rotated query meets the rotated key, their dot product contains a phase difference related to the positions.

ALiBi, or Attention with Linear Biases, takes a different route. It adds a head-specific penalty based on the distance between query and key positions. A token ten positions away can therefore receive a different score from a token two positions away, even when their content vectors are equally similar.

The senior-level nuance

A causal mask is not the same thing as positional encoding. In an autoregressive decoder, the mask prevents position i from attending to later positions. The triangular mask gives the model a weak structural sense of before and after, and each position has a different prefix size.

So a causal transformer without explicit positional embeddings may recover some order information. Saying it becomes a perfect bag of words is too strong. But the mask does not directly tell the model that two tokens are three positions apart, nor does it provide a clean reusable coordinate for bidirectional attention. It is a partial architectural signal, not a general replacement for positional information.

There are also tasks where position should not matter. If the input is an unordered set of products, points, or database records, adding arbitrary sequence positions can be harmful because two equivalent inputs would receive different representations. In those cases, permutation-invariant or permutation-equivariant attention may be exactly what you want.

The choice also depends on context length. A learned absolute table trained for 512 positions does not naturally know what position 513 means. A production request longer than that may raise an index or shape error, depending on the implementation. RoPE and sinusoidal encodings can generate positions beyond the training range, but extrapolation is not magic. The model still needs to have learned attention patterns that remain sensible at those distances.

One practical failure mode appears when position IDs are mishandled during generation. If a key-value cache resets position IDs to zero at every chunk, the model may treat the beginning of each chunk as the beginning of the document. The first few tokens can look normal, while long-context answers degrade around chunk boundaries or references drift toward the wrong section. Position IDs, padding conventions, attention masks, and cache offsets must agree with the model’s positional scheme.

What they’ll ask next

“Can a transformer work without positional encoding?”

Yes, when the input is genuinely unordered and the desired function should not change under permutation. For ordinary language, explicit position is usually needed. A causal mask can provide limited ordering information, but it is not a robust substitute.

“Why are RoPE and ALiBi often preferred over learned absolute embeddings?”

They represent relative distance more directly, which matches many language patterns: nearby tokens often matter differently from distant tokens. They also avoid a simple learned lookup-table limit. The trade-off is that long-context behavior still depends on training distribution and implementation details; neither method guarantees arbitrary length generalization.

“Why add a positional vector instead of concatenating it?”

Addition keeps the model width unchanged, so the existing projection matrices can process content and position together without expanding every layer. Concatenation is possible, but it increases the input width and parameter costs and requires a different architecture. Addition is a design choice, not a mathematical requirement.

Say this in the interview

A transformer needs positional information because self-attention is content-based and permutation-equivariant; adding absolute or relative position lets it represent order and distance, while the best scheme depends on whether the task is ordered and how far beyond the training context it must generalize.