Positional encodings (sinusoidal & RoPE)

If you shuffle the tokens in an attention layer’s input, the output shuffles the same way and nothing changes. Attention has no position-awareness on its own — it treats the input as a set, not a sequence. That’s a problem for language (“dog bites man” ≠ “man bites dog”), so transformers inject position information explicitly.

The original recipe: sinusoidal encodings

The 2017 Attention Is All You Need paper added a fixed position-dependent vector to each token embedding:

PE(pos, 2k)   = sin(pos / 10000^(2k/d))
PE(pos, 2k+1) = cos(pos / 10000^(2k/d))

Even dims get sines, odd dims get cosines, each at a different frequency. Different positions get different “fingerprints.” These vectors are fixed — not trained — but the model learns to read them.

import numpy as np

def sinusoidal_pe(max_len, d_model):
    pe = np.zeros((max_len, d_model))
    pos = np.arange(max_len)[:, None]
    i = np.arange(d_model)[None, :]
    angle = pos / (10000 ** (2 * (i // 2) / d_model))
    pe[:, 0::2] = np.sin(angle[:, 0::2])
    pe[:, 1::2] = np.cos(angle[:, 1::2])
    return pe

pe = sinusoidal_pe(max_len=20, d_model=16)
print("PE shape:", pe.shape)
# A simple ASCII heatmap of the pattern (16 dims x 20 positions)
print("Pattern (rows=position, cols=dim, '+' = positive, '-' = negative):")
for row in pe:
    print("".join("+" if v > 0.3 else "-" if v < -0.3 else "." for v in row))

PE shape: (20, 16)
Pattern (rows=position, cols=dim, '+' = positive, '-' = negative):
.+.+.+.+.+.+.+.+
++++.+.+.+.+.+.+
+-++.+.+.+.+.+.+
.-++.+.+.+.+.+.+
--++++.+.+.+.+.+
-.+.++.+.+.+.+.+
.++-++.+.+.+.+.+
+++-++.+.+.+.+.+
+.+-++.+.+.+.+.+
+-.-++.+.+.+.+.+
--.-++++.+.+.+.+
-.--++++.+.+.+.+
-+--++++.+.+.+.+
++--+.++.+.+.+.+
+.-.+.++.+.+.+.+
+--.+.++.+.+.+.+
.--++.++.+.+.+.+
-.-++.++.+.+.+.+
-+-++.++.+.+.+.+
.+.++-++.+.+.+.+

Each position has its own unique pattern of +/−/·, and adjacent positions have similar patterns (so “close in position” → “close in encoding”). The model can decode relative position from the difference between two PE vectors.

Each row is a position; reading down a column, you can see low dimensions (left) oscillate fast while high dimensions (right) barely change across these 20 positions — that mix of frequencies is what makes every position’s fingerprint unique. Sinusoidal PEs were used in BERT, GPT-2, the original T5, and most 2018–2021 transformers.

The problems with sinusoidal PEs

Two issues showed up at scale:

1. Position is mixed with content. The model has to disentangle “what is this token?” from “where is it?” inside every attention layer. Works, but imperfectly.
2. They don’t extrapolate. Train at length 2048, use at length 8192, and accuracy drops sharply.

RoPE — Rotary Positional Embeddings

Rotary positional embeddings solve both problems with a clever trick: instead of adding position info to embeddings, rotate the Q and K vectors by an angle proportional to position before computing attention scores.

The intuition: rotate token m’s query by angle m·θ, rotate token n’s key by angle n·θ. Their dot product depends on the angle difference (m - n)·θ — purely relative position. The model gets relative-position information for free, without any added embedding.

import numpy as np

def rope_freqs(d_head, base=10000):
    # Frequencies per pair of dimensions
    # d_head must be even
    half = d_head // 2
    return 1.0 / (base ** (np.arange(half) * 2 / d_head))

def apply_rope(x, position):
    # x: (d_head,), position: integer
    # Rotate pairs of dims: (x_2k, x_2k+1) by angle position * freq_k
    d_head = x.shape[-1]
    freqs = rope_freqs(d_head)
    angles = position * freqs        # (d_head/2,)
    cos, sin = np.cos(angles), np.sin(angles)
    x_pairs = x.reshape(-1, 2)
    x0, x1 = x_pairs[:, 0], x_pairs[:, 1]
    rotated = np.stack([x0 * cos - x1 * sin, x0 * sin + x1 * cos], axis=-1)
    return rotated.reshape(d_head)

# Example: same vector, different positions -> different rotations
v = np.array([1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0])
print("position 0:", apply_rope(v, 0).round(3))
print("position 1:", apply_rope(v, 1).round(3))
print("position 5:", apply_rope(v, 5).round(3))

# Crucially: dot product depends on relative position
v_q_at_3 = apply_rope(v, 3)
v_k_at_5 = apply_rope(v, 5)
v_q_at_10 = apply_rope(v, 10)
v_k_at_12 = apply_rope(v, 12)
print("\n<q@3, k@5> =", v_q_at_3 @ v_k_at_5)
print("<q@10, k@12> =", v_q_at_10 @ v_k_at_12)
print("Both have a position gap of 2 -> same dot product!")

position 0: [1. 0. 1. 0. 1. 0. 1. 0.]
position 1: [0.54  0.841 0.995 0.1   1.    0.01  1.    0.001]
position 5: [ 0.284 -0.959  0.878  0.479  0.999  0.05   1.     0.005]

<q@3, k@5> = 2.563717747961344
<q@10, k@12> = 2.5637177479613436
Both have a position gap of 2 -> same dot product!

Two pairs with the same relative position have the same dot product after RoPE, even though their absolute positions differ. That’s the magic — relative position falls out of the geometry automatically.

Other variants worth knowing

• ALiBi: adds a position-dependent bias directly to attention scores. Used in BLOOM, MPT. Strong extrapolation.
• Learned: a trainable vector per position, capped at training length. BERT, GPT-2. Doesn’t extrapolate.
• NoPE: skip positional encoding; rely on causal mask for order. Works for small models, doesn’t scale.

Where in the network it goes

RoPE’s “rotate Q and K at every layer” approach is more compute than “add once,” but it’s still cheap and the benefits are large.

In one breath

• Attention is permutation-equivariant — shuffle the tokens and the output shuffles the same way, so order has to be injected explicitly.
• Sinusoidal PEs add a fixed sine/cosine fingerprint to each embedding (one frequency per dimension pair). Simple, but position is mixed into content and it doesn’t extrapolate past the training length.
• RoPE instead rotates Q and K by an angle proportional to position, so their dot product depends only on the relative gap (m − n)·θ — applied at every attention layer, not added once.
• That relative, geometric encoding is why RoPE is the default in nearly every modern LLM (Llama, Mistral, Qwen, DeepSeek) and stretches to long context with tricks like NTK-aware scaling; ALiBi is the other strong-extrapolation option.

Quick check