What are positional encodings and why are they needed in transformers?

How to think about it

Positional encodings are needed because self-attention—the operation that lets each token compare itself with every other token—has no built-in first, second, or distance when it receives only token content. They inject absolute position or relative distance, either by adding a vector to each token representation or by changing attention scores, so dog bites man can mean something different from man bites dog.

Why self-attention needs help

Imagine giving a transformer the tokens dog, bites, and man, but not telling it where each token appeared. The model receives three vectors containing information about those words. It can learn that dog and man are nouns and that bites is a verb. It cannot tell which noun came first.

That matters because word order changes meaning. The dog bites the man and The man bites the dog use almost the same vocabulary, but the subject and object have swapped.

A self-attention layer creates three vectors for each token: a query, which describes what the token is looking for; a key, which describes what it offers to other tokens; and a value, which contains the information that can be passed along. The attention score between token i and token j is roughly:

score(i, j) = query_i · key_j / sqrt(d_k)

The score depends on the token representations and their learned projections. Without positional information, it does not depend on whether token j was immediately before token i, ten places away, or at the beginning of the sentence.

Strictly speaking, unmasked self-attention without positions is permutation-equivariant. That means reordering the inputs reorders the outputs in the same way. It does not mean every output is literally unchanged. But the practical consequence is the important part: the layer has no basis for treating “first” and “third” as different roles.

Common mistake: “Self-attention is permutation-invariant” is useful shorthand, but technically a sequence of per-token outputs is permutation-equivariant. A pooled output, such as a sum or mean over tokens, is permutation-invariant. In both cases, content alone does not provide order.

Adding more content-only attention layers does not fix this. The feed-forward part of a transformer processes each position independently, so a stack of such layers preserves the same symmetry.

A causal mask gives a decoder a weak ordering cue: position 5 may attend to earlier positions, while position 4 cannot attend to position 5. But the mask says which interactions are allowed; it does not directly represent distance, direction, or an explicit coordinate for every token. Bidirectional encoder attention has no causal mask at all. That is why practical transformers use a positional mechanism even when they also use masking.

The mechanism

Let e_i be the embedding for token i, and let p_i be its positional representation. The simplest construction is:

h_i = e_i + p_i

The transformer receives h_i, not just e_i. The same word therefore produces a different input vector at different positions. Its query and key become position-dependent after the usual learned projections, so attention can learn patterns such as:

• attend strongly to the previous token;
• look for a noun several positions to the left;
• treat a token after a negation differently from the same token elsewhere;
• preserve information from the beginning of a long passage.

The position vector does not tell the model what the position means. It only supplies a usable signal. Training teaches the model how to use it.

There is another family of designs that does not add p_i to the token embedding. Instead, it changes the attention logit, the score before softmax normalization:

score(i, j) = content_score(i, j) + position_bias(i, j)

A bias based on i - j can tell the model that nearby tokens matter more, or that “before” and “after” are different relationships.

A concrete numerical example

Take a four-dimensional embedding for the word bank:

e_bank = [0.2, 0.7, -0.1, 0.4]

A classic sinusoidal encoding uses alternating sine and cosine functions:

PE(pos, 2i) = sin(pos / 10000^(2i / d_model))

PE(pos, 2i + 1) = cos(pos / 10000^(2i / d_model))

With d_model = 4, position zero has:

p_0 = [0, 1, 0, 1]

So the input becomes:

h_0 = [0.2, 1.7, -0.1, 1.4]

At position one, the encoding is approximately:

p_1 = [0.8415, 0.5403, 0.0100, 0.99995]

The same word now becomes:

h_1 = [1.0415, 1.2403, -0.0900, 1.39995]

The first coordinate is not simply a “position number,” and no single coordinate means “this is token one.” Position is distributed across several dimensions, with different frequencies. The short and long waves give the model signals at different distance scales.

These encodings are absolute: position zero gets one vector, position one gets another, and so on. A learned absolute encoding works similarly, except each p_i is a trainable lookup vector rather than a fixed mathematical function.

The main positional schemes

Scheme	How it works	Main trade-off
Learned absolute	Add a trainable vector for each position	Simple, but normally tied to a maximum trained length
Sinusoidal absolute	Add a fixed sine and cosine pattern	Can calculate new positions, but longer-context quality is not guaranteed
Relative bias	Add a bias based on the distance between two positions	Directly represents distance; bucketing and direction choices matter
RoPE	Rotate queries and keys by position-dependent angles	Efficient and strong for language models; long-context scaling needs care
ALiBi-style bias	Penalize attention according to distance	Simple distance preference, but imposes a particular inductive bias

RoPE, or Rotary Position Embedding, deserves special attention because it is common in decoder-only language models. It rotates pairs of query and key dimensions by an angle determined by position. If R_i is the rotation for position i, the score uses terms like:

(R_i q_i) · (R_j k_j)

Because rotations compose by angle difference, this interaction depends on the relative offset between i and j, not merely on two unrelated position vectors. RoPE therefore carries absolute phase into the query and key while making their dot product sensitive to relative position. That is why calling it simply “an absolute encoding” misses the useful part.

The senior-level nuance

There is no universally best positional encoding. The right choice depends on the architecture, training context length, target context length, and whether order is genuinely meaningful.

Learned absolute embeddings are straightforward when the model always operates within a known limit, such as sequences of at most 512 tokens. They become awkward when the product later needs 16,000 tokens: positions beyond the learned table do not exist unless the model is modified and validated.

Sinusoidal encodings avoid that lookup-table limit, but being computable at a new position does not mean the model will generalize there. A model trained on short sequences may still behave badly on long ones.

Relative schemes often make the desired relationship more explicit. The model can distinguish “one token before” from “one token after,” and it may group very distant positions into buckets. That can improve efficiency or generalization, but bucketing throws away some exact distance information.

RoPE also has a real long-context trade-off. Its frequencies and rotations were chosen during training. Extending the context window can cause phase patterns that the model rarely saw, so quality may deteriorate near the end even when memory and latency look healthy. Position interpolation or frequency scaling can help, but those are model-specific changes that must be evaluated rather than assumed safe.

Do not casually swap positional schemes after pretraining. The attention weights learned one particular geometry. Replacing RoPE with learned absolute vectors, or changing its frequency configuration, changes what query-key dot products mean. That usually requires retraining, careful conversion, or strong regression tests.

There is also a case where positional encoding is undesirable: unordered data. If a model is processing a set of products where row order has no meaning, assigning position zero to one product and position one to another creates a false distinction. In that setting, preserving permutation invariance may be the correct design.

What fails in production

A common failure is position and cache state getting out of sync during generation. A KV cache, which stores prior keys and values so the model does not recompute the whole prompt for every new token, must use the same position convention as the initial prompt.

For a 512-token prompt, the first generated token normally receives the next position, such as position 512 under zero-based indexing. If an inference path accidentally labels it position zero, the model still runs. Latency may look perfectly normal. The output often becomes repetitive, incoherent, or noticeably worse after the first generated token.

Long-context extension has a different symptom: short prompts pass all tests, but answers become garbled or retrieval accuracy falls for information near the context limit. That points to positional extrapolation or attention behavior, not necessarily a memory problem.

What they’ll ask next

“Does the causal mask remove the need for positional encodings?”

No. A causal mask supplies directional visibility, which is some structural information about order. It does not provide the rich, learned representation of distance and position that a decoder normally needs, and it cannot help a bidirectional encoder. Most causal transformers use both a causal mask and a positional scheme.

“If RoPE uses a position-dependent rotation, why is it called relative?”

The rotation is assigned from each token’s absolute index, but the query-key dot product depends on the difference between the two rotation angles. In effect, attention can sense how far apart two tokens are and which one comes first. That is the useful relative-position property.

“Are sinusoidal encodings better than learned encodings?”

Not by default. Learned encodings are often convenient and effective within a fixed context length. Sinusoidal encodings have no learned position table and can be evaluated beyond the training range, but that does not guarantee reliable extrapolation. The training distribution and the deployment context matter more than the slogan.

Say this in the interview

“Self-attention compares token content but has no inherent order, so positional information is required; it can be added as absolute learned or sinusoidal vectors, or injected into attention through relative biases or schemes such as RoPE, with the choice driven by context length, architecture, and extrapolation needs.”