What do the query, key, and value vectors represent in attention?

How to think about it

The direct answer

A query represents what a token is looking for, a key represents what a token offers as a possible match, and a value represents the information that token actually contributes. Attention compares a query with all eligible keys, converts those compatibility scores into weights, and uses the weights to blend the corresponding values.

The useful mental model is a soft information lookup: the query asks, the keys decide what matches, and the values provide the retrieved content.

The mechanism: a learned soft lookup

Suppose a sequence has n token positions. Position i is the token currently producing an output. Position j is a candidate source of information.

The input to an attention layer is a matrix X containing one hidden vector per token. In self-attention, the layer creates three new matrices with learned linear projections:

Q = X W_Q
K = X W_K
V = X W_V

Q, K, and V have one row per token, but they serve different jobs.

For position i, the query vector Q_i is compared with every candidate key K_j. Their dot product measures compatibility:

score(i, j) = Q_i · K_j / sqrt(d_k)

Here, d_k is the number of components in each key and query vector. The division by sqrt(d_k) keeps dot products from becoming too large as the vectors get wider.

The model then applies softmax across the candidate positions:

a_ij = softmax over j(score(i, j))

The resulting a_ij values are non-negative and sum to one for a fixed query position i. Finally, the attention output at position i is:

output_i = sum over j of a_ij V_j

So the key influences how much attention a source receives. The value determines what information arrives from that source.

That separation is the whole point. A token can be a good match for a query without contributing the same features that made it a good match.

A concrete example

Consider this sentence:

The animal did not cross the street because it was tired.

At the position containing it, the model may need to resolve what the pronoun refers to. A query produced at that position could encode a requirement such as “find a plausible earlier entity that can be the subject of this clause.”

The key for animal might advertise features related to an entity or possible antecedent. The key for street might advertise location-related features. The value for animal carries the representation that the model wants to retrieve if animal is relevant. The value for street carries different content.

Those descriptions are intuition, not a guaranteed dictionary for the vectors. The model does not store a tiny English sentence inside each vector. It learns whatever numerical features help reduce its training loss.

Now use a small numerical example. Suppose one query has three eligible keys, and the already-scaled query-key scores are:

[2.0, 1.0, 0.0]

Softmax turns them into approximately:

[0.6652, 0.2447, 0.0900]

Suppose the corresponding values are two-dimensional:

V_1 = [10, 0]
V_2 = [0, 10]
V_3 = [4, 4]

The output is:

0.6652 [10, 0] + 0.2447 [0, 10] + 0.0900 [4, 4]
= [7.0125, 2.8074]

In PyTorch:

import torch

scores = torch.tensor([2.0, 1.0, 0.0])
weights = torch.softmax(scores, dim=0)

values = torch.tensor([
    [10.0, 0.0],
    [0.0, 10.0],
    [4.0, 4.0],
])

print(weights)
print(weights @ values)

The output is approximately:

tensor([0.6652, 0.2447, 0.0900])
tensor([7.0125, 2.8074])

This is not a hard choice of the first value. It is a weighted mixture. The highest-scoring key receives most of the weight, but the other values still contribute.

The result is a latent vector, not necessarily a readable word or phrase. Later layers combine it with the original representation through residual connections and further transformations.

Why use three separate projections?

The model could use the same vector for matching and content, but that would force one representation to do two different jobs.

A useful analogy is a customer-support search system. The search index might use a compact representation of product category, issue type, and urgency to decide which article matches. The article body is a different representation: it contains the instructions to show the customer. Matching features and delivered content overlap, but they are not identical.

Attention learns the same separation:

• W_Q maps a token into the space that expresses what it needs.
• W_K maps a token into the space that makes its relevance discoverable.
• W_V maps a token into the space containing useful transferable information.

If the model wanted position i to find positions describing grammatical subjects, W_Q and W_K could learn features useful for that comparison. W_V could preserve entity information that should be passed onward once a subject is found.

In multi-head attention, each head has its own projections, at least conceptually. One head may learn a pattern related to subject-verb relationships, another may track nearby positions, and another may move information about entities. The implementation may combine projection operations into larger matrices for efficiency, but the three logical roles remain.

There is an important qualification: separate projections do not force these meanings. They merely give the model the freedom to discover them.

Warning: In self-attention, queries, keys, and values usually all originate from the same input sequence, but they are not the same vectors. They are different learned projections of that sequence. And “query means need” is an interpretation that often helps humans reason about the mechanism, not a semantic rule enforced by the architecture.

The distinction between self-attention and cross-attention

In self-attention, queries, keys, and values come from the same sequence. Every token creates a query, key, and value from its own hidden state. The query at one position looks across the keys from other positions, subject to attention masks.

In cross-attention, the sources differ. For example, in an encoder-decoder translation model:

• Queries come from the decoder’s current hidden states.
• Keys and values come from the encoder’s output for the source sentence.

If the decoder is generating the French translation of an English sentence, its query asks something like “which source information is useful for the next French token?” The encoder keys make source positions matchable, and the encoder values carry the source information to retrieve.

This is why saying “the query comes from the current token and the key and value come from every token” is correct for ordinary self-attention but incomplete as a general statement about transformers.

The nuance that earns the senior signal

First, attention weights are not guaranteed explanations. If a head assigns a weight of 0.8 to one token, that tells you the token strongly influenced that head’s weighted sum. It does not prove that the model “reasoned because of” that token in a human-interpretable sense. Multiple heads and later layers can transform, cancel, or amplify the result.

Second, query-key dot products are not automatically semantic similarity scores. They are not necessarily cosine similarities because the vectors are not required to have unit length. The learned spaces are task-specific, and a large dot product may represent a syntactic, positional, or highly specialized pattern rather than similar topic or meaning.

Third, softmax weights sum to one, but the values themselves may contain negative components. The output is a convex combination of value vectors in the mathematical sense of its weights, not necessarily a positive or human-readable blend of concepts.

Finally, masking changes what “all keys” means. A padding mask prevents attention to nonexistent input. A causal mask prevents a decoder position from reading future tokens. The model can only retrieve from keys that remain visible after masking.

A failure mode you can recognize in practice

A common implementation bug is applying softmax along the wrong dimension. For a score matrix with one row per query and one column per candidate key, normalization must happen across the key columns for each query.

The usual pattern is conceptually:

scores = Q @ K.transpose(-2, -1) / (d_k ** 0.5)
weights = torch.softmax(scores, dim=-1)
output = weights @ V

If softmax is applied across the query dimension instead, columns sum to one rather than rows. Each query no longer receives a properly normalized distribution over source positions.

The first symptom may be surprisingly unstable behavior: changing the batch padding or sequence length changes outputs that should be unrelated. In a causal language model, a missing or incorrect causal mask can produce an even more suspicious result: training loss looks excellent because the model can read future tokens, but generated text falls apart when future tokens are unavailable.

What they’ll ask next

Why divide by the square root of d_k?
If query and key components have roughly stable variance, the variance of their dot product grows with the vector width. Large scores make softmax nearly one-hot, which produces tiny gradients for the losing positions. Dividing by sqrt(d_k) keeps the scores in a more useful range during training.

How are Q, K, and V different in cross-attention?
The queries come from the sequence requesting information, while keys and values come from the sequence supplying it. In machine translation, decoder states produce queries and encoder states produce keys and values.

Does a high attention weight prove the model used a token for reasoning?
No. It proves that the token’s value had a large contribution to that particular attention output. Attention is part of the computation, not a complete causal explanation of the model’s decision.

Say this in the interview

“Queries express what a position is looking for, keys express what each position offers for matching, and values carry the information retrieved; attention scores query-key compatibility, softmaxes those scores, and returns a weighted sum of the values.”