Explain self-attention and the roles of the Query, Key, and Value vectors.

How to think about it

Self-attention lets each token—a word or word-piece represented by a vector, meaning a list of learned numbers—look at other tokens in the same sequence and build a context-aware representation. For one token, the mechanism compares its Query with every Key, converts those scores into weights with softmax, and takes a weighted sum of the corresponding Values. Query, Key, and Value are separate learned projections of the original token representation.

Why three vectors?

Suppose the input representation for token i is x_i. The model turns it into three vectors:

Vector	Plain-English role	What it controls
Query, q_i	What this token is looking for	Which other tokens seem relevant
Key, k_i	What this token offers as a match	Whether another token should attend to it
Value, v_i	The information this token contributes	What gets copied into the output

The names come from a retrieval-system metaphor. A query asks for something. A key describes what can be found. A value is the payload returned after a match. The Value is not a “true value” in the statistical sense, and the Query and Key are not human-readable labels. They are learned numerical representations.

A linear projection is multiplication by a learned weight matrix, optionally with a bias. For every input token, the model computes something like q_i = x_i W_Q, k_i = x_i W_K, and v_i = x_i W_V. The matrices W_Q, W_K, and W_V are adjusted during training so that useful relationships produce useful outputs.

The separation matters. A token can use one set of features to advertise that it is a good match, while sending a different set of features as its payload. If Query, Key, and Value were all the same vector, the model would have much less freedom to distinguish “should I attend here?” from “what information should I receive from here?”

The mechanism

Let X contain the representations of all n tokens. Self-attention computes:

Q = XW_Q

K = XW_K

V = XW_V

The complete attention operation is:

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k))V

Here, d_k is the number of components in each Key. For one query q_i and one key k_j, the model first calculates a dot product: multiply matching coordinates and add them. That gives a compatibility score:

score(i, j) = q_i · k_j / sqrt(d_k)

A high score means the Query and Key point in similar directions in the learned space. The softmax then turns all scores for query i into non-negative weights that add up to one. Finally, the model multiplies each Value by its weight and adds the results.

That last step is the important one. The model does not simply select the single best token. It usually creates a mixture. A token might receive 70 percent of its information from one location, 20 percent from another, and 10 percent from several others.

The division by sqrt(d_k) prevents a practical problem. If the Query and Key contain many roughly independent components, their dot products tend to grow in magnitude as the dimension grows. Large scores make softmax extremely peaked: one position gets almost all the weight, the others get almost none, and useful gradients become small. Scaling keeps the scores in a more manageable range.

Because the whole operation is differentiable, nobody hand-codes rules such as “pronouns should attend to nouns.” Training adjusts the projection matrices so that attention patterns that help the task reduce the loss. The model discovers the matching behavior.

A concrete numerical example

Consider the sentence:

The animal did not cross the street because it was tired.

Focus on the token it. To keep the arithmetic visible, consider two candidate tokens: animal and street.

Imagine that the Query for it is:

q_it = [1, 1]

The Keys are:

k_animal = [1, 1]

k_street = [1, 0]

The raw dot products are therefore 2 and 1. Let the Key dimension be d_k = 2. After scaling by sqrt(2), the scores become approximately 1.414 and 0.707.

Softmax gives approximately:

• animal: 0.67
• street: 0.33

Now suppose the Values are:

v_animal = [10, 2]

v_street = [0, 8]

The output for it is:

0.67[10, 2] + 0.33[0, 8] = [6.7, 3.98]

The resulting representation contains more of the information carried by animal than the information carried by street. In a real model, the vectors have hundreds or thousands of dimensions, and the useful relationship might concern grammar, topic, code structure, or factual context rather than pronoun resolution. But the calculation is the same.

Common misconception: attention does not perform a symbolic lookup and does not necessarily choose one antecedent. It produces a weighted mixture. Also, an attention weight is not automatically a complete explanation of the model’s decision. Later layers, other heads, residual connections, and feed-forward networks all affect the final result.

How self-attention fits into a Transformer

It is called self-attention because the same input sequence supplies the Queries, Keys, and Values. An encoder can let every token attend to every other token. A decoder generating text normally uses causal attention, where a mask prevents a position from attending to future positions. The masked scores are set to negative infinity before softmax, so their weights become zero.

That mask is not optional. During generation, the future tokens do not exist yet. If training lets the model see them, the model can appear impressively accurate by reading the answer in advance.

Transformers usually use multi-head attention. Each head has its own Query, Key, and Value projections. One head may learn a useful syntactic relationship while another tracks a long-range reference or a formatting pattern. The head outputs are concatenated and passed through another learned projection. “One head learns subject-verb agreement” is a possible interpretation, not a promise; heads are often messy and can split or share responsibilities across layers.

Self-attention also needs positional information. Without it, the operation has no built-in notion that dog bites man differs from man bites dog; permuting the input tokens simply permutes the outputs in the same way. Transformers add position through methods such as learned positional embeddings, sinusoidal encodings, or rotary position embeddings.

The senior-level trade-off

Full self-attention compares every token pair. Its standard cost grows as O(n^2) with sequence length n. A sequence of 4,096 tokens creates 16,777,216 query-key pairs per head in one layer. If the score matrix is materialized in 16-bit floating point, that matrix alone is about 32 MiB, before storing gradients, projections, and other activations.

Techniques such as FlashAttention reduce peak memory and improve memory access by computing attention in tiles, but exact full attention still has quadratic pairwise work. For very long documents or endless event streams, local attention, sparse attention, retrieval, chunking, recurrent designs, or state-space models may be better choices. The choice depends on whether every token genuinely needs direct access to every other token, and on the latency and memory budget.

At autoregressive inference, a KV cache stores the Keys and Values for previous tokens so the system does not recompute them at every generation step. This reduces repeated computation, but the cache itself grows with context length, model depth, and the number of attention heads. It trades computation for memory rather than making long context free.

A failure mode worth mentioning

A missing or incorrectly applied causal mask often shows up first as suspiciously low validation loss followed by poor free-running generation. The model learned with access to future target tokens, then loses that shortcut at inference time.

A missing padding mask produces a different symptom: the same text can receive different outputs depending on how much padding surrounds it in a batch. Padded positions are not meaningful content, so allowing them into the softmax changes the mixture the model sees.

What they’ll ask next

Why use sqrt(d_k) in the score?
Dot products tend to grow in magnitude as the Key dimension grows. Without scaling, softmax saturates around one position and gradients become less useful. Dividing by sqrt(d_k) keeps the logits better behaved.

How is self-attention different from cross-attention?
In self-attention, Q, K, and V come from the same sequence. In cross-attention, Queries come from one sequence and Keys and Values come from another—for example, a decoder’s Queries attending to an encoder’s output.

Why not use one attention head?
One head must express all relationships in one learned compatibility space. Multiple heads provide several independently projected spaces, allowing different kinds of relationships to be represented at the same layer, at the cost of more parameters and computation.

Say this in the interview

“Self-attention lets each token query every other token: Queries express what it is looking for, Keys determine what matches, and Values carry the information that gets mixed into the output; the scores are scaled dot products normalized by softmax.”