What are positional encodings and why are they needed in transformers?

For ML Engineer research-engineer AI / LLM Engineer

The short answer

Self-attention is permutation-invariant and has no inherent notion of token order, so positional encodings inject information about each token's position. They can be fixed sinusoidal functions or learned embeddings added to inputs, or relative schemes like RoPE that modulate attention by relative distance.

How to think about it

Self-attention is permutation-invariant and has no inherent notion of token order, so positional encodings inject information about each token’s position. They can be fixed sinusoidal functions or learned embeddings added to inputs, or relative schemes like RoPE that modulate attention by relative distance.

Learn it properly Positional encodings & RoPE

Keep practising

Why does a transformer need positional encoding? What does self-attention actually compute, and why is it useful? Walk me through the transformer encoder architecture block by block. What is the difference between encoder-only, decoder-only, and encoder-decoder transformer architectures? What is the difference between encoder-only, decoder-only, and encoder-decoder transformer architectures?

All Deep Learning questions

Explore further

The Transformer Architecture Multi-head attention Tokenization & BPE

Positional Encoding Transformer Self-Attention Encoder