How do temperature, top-k, and top-p sampling control LLM generation?

How to think about it

The direct answer

Temperature changes how sharply the model’s probability distribution is concentrated, while top-k and top-p remove unlikely token candidates before one is sampled. Lower temperature, smaller k, and smaller p usually make output more predictable; higher values preserve more alternatives and therefore more variation, but none of these settings makes an answer more factual or structurally valid.

Why these controls exist

An LLM generates text one token at a time, where a token is a small piece of text such as a word, part of a word, or punctuation. At each step, the model produces a logit, an unnormalised score, for every token in its vocabulary. The model then converts those scores into probabilities with softmax:

p_i = exp(z_i) / sum_j exp(z_j)

Here, z_i is the logit for token i, and p_i is its probability. The probabilities add up to one.

There are two broad ways to choose the next token:

• Greedy decoding chooses the single token with the highest probability.
• Sampling draws a token randomly according to the probability distribution.

Suppose the model gives "the" a probability of 0.60, "a" a probability of 0.25, and "zebra" a probability of 0.001. Sampling will usually choose "the", but it can choose something else. After that token is appended to the context, the model calculates a completely new distribution for the next position.

Temperature changes the shape of that distribution. Top-k and top-p change which candidates are allowed to participate in the draw.

Temperature changes the sharpness

Temperature applies a positive scalar T to the logits before softmax:

p_i(T) = softmax(z_i / T)

At T = 1, the original distribution is unchanged.

At T < 1, dividing the logits by a smaller number makes their differences larger. The leading tokens receive more probability, and the tail receives less. At T > 1, the differences are compressed, so probability spreads across more tokens.

Temperature does not change the ranking of tokens as long as T is positive. If token A has a higher logit than token B, it remains higher after dividing both logits by the same positive temperature. It changes their relative probabilities, not which one is ranked first.

Consider this distribution at T = 1:

Token	T = 1	T = 0.5	T = 2
"the"	0.460	0.708	0.304
"a"	0.250	0.209	0.224
"this"	0.120	0.048	0.155
"your"	0.080	0.021	0.127
"one"	0.050	0.008	0.100
"zebra"	0.040	0.005	0.090

The last two columns are approximate results from applying temperature to the first column. At T = 0.5, "the" becomes much more dominant. At T = 2, less likely continuations become plausible.

The mathematical formula is not normally used with exactly zero because division by zero is undefined. Many APIs treat temperature zero as a special request for greedy decoding; others reject it or impose their own behaviour. Check the provider’s documentation rather than assuming that 0 means precisely the same thing everywhere.

Temperature alone never removes a token. Even at a low temperature, a very unlikely token technically remains possible unless another decoding rule filters it out.

Top-k keeps a fixed number of candidates

Top-k sampling sorts tokens by probability, keeps the k most likely, discards the rest, and renormalises the survivors so that they again sum to one.

Using the distribution above:

• With top_k = 3, only "the", "a", and "this" remain.
• Their original total probability is 0.46 + 0.25 + 0.12 = 0.83.
• After renormalisation, their probabilities become approximately 0.554, 0.301, and 0.145.

The model is no longer sampling from the full vocabulary. It is sampling from those three tokens.

top_k = 1 is effectively greedy decoding, apart from tie-handling and provider-specific details. A very large k, at least as large as the vocabulary, does not meaningfully restrict the distribution.

The strength of top-k is predictability: it imposes a clear hard ceiling on the number of candidates. Its weakness is that the ceiling is fixed even when the model’s uncertainty changes. In a confident context, the best three tokens may contain nearly all useful probability. In an ambiguous context, the correct continuation might be ranked fourth or fiftieth and get removed.

Top-p keeps a variable probability mass

Top-p, also called nucleus sampling, sorts tokens from most to least probable and keeps the smallest prefix whose cumulative probability reaches p.

This is not the same as keeping every token whose individual probability exceeds p. The cutoff is based on the running total.

With the same distribution:

Setting	Tokens retained	Original mass
top_p = 0.70	"the", "a"	0.71
top_p = 0.85	"the", "a", "this", "your"	0.91

For top_p = 0.70, the first token alone contributes 0.46, which is not enough. Adding the second reaches 0.71, so the nucleus contains two tokens. Their renormalised probabilities are about 0.648 and 0.352.

For top_p = 0.85, the first three tokens total only 0.83. The fourth token must be included, taking the retained mass to 0.91.

That changing candidate count is the central advantage of top-p. When the model is confident, the nucleus may contain only a few tokens. When the model is uncertain, it can expand to include many plausible continuations. The trade-off is less direct control than top-k: top_p = 0.90 might retain two tokens at one position and thirty at another.

A value of p = 1 normally keeps the whole distribution. A value close to one does not necessarily mean many tokens will survive. If the leading token already has probability 0.995, even a high nucleus threshold can leave only that token.

How they work together

A common conceptual decoding pipeline is:

logits → temperature → top-k or top-p filtering → renormalisation → random draw

The exact order of top-k and top-p can vary by implementation. If both are enabled, the final candidate set is more restricted than either setting alone, and the order can matter because top-p calculates a cumulative probability over whichever tokens are present at that stage. Temperature matters especially for top-p: it changes the probabilities and therefore can change where the cumulative cutoff falls. It usually does not change which tokens top-k ranks first.

The practical choice depends on the job:

• For extraction or classification, low temperature or greedy decoding reduces accidental variation. Validation is still necessary.
• For customer-support prose, moderate randomness can avoid robotic repetition while keeping answers stable.
• For brainstorming, a higher temperature or broader nucleus can expose less obvious continuations.
• For code or JSON, sampling settings are not a substitute for grammar constraints, structured decoding, parsing, and retries.

The senior-level point is that randomness and correctness are different dimensions. A low-temperature model can be confidently wrong because it is selecting the model’s most likely wrong answer. A high-temperature model can occasionally discover a better phrasing, but it also gives malformed or irrelevant tokens more opportunity. The model’s probabilities are not a calibrated truth meter.

Common misconception: setting temperature = 0.1 does not guarantee valid JSON, factual answers, or identical output. It only makes the probability distribution more concentrated.

A failure mode you would see in production

Suppose a data-extraction service repeatedly returns a response that begins with valid JSON but occasionally ends after "city": "Paris", without a closing brace. The team lowers temperature and sets a very small top-p, expecting the problem to disappear. Instead, the service produces the same malformed shape more consistently.

The decoding settings did exactly what they were asked to do: they selected among likely next tokens. They did not impose a grammar. If the model’s most likely continuation is still a closing quote, a comma, or an unfinished field, lower randomness merely makes that mistake more repeatable.

The fix is to use structured or constrained decoding when the provider supports it, parse the result, validate it against a schema, and retry or surface an error when validation fails. Sampling controls can reduce variation; they cannot replace a contract.

What they’ll ask next

Does lowering temperature always make output deterministic?
No. It reduces randomness but does not remove it unless decoding becomes greedy or the system provides a deterministic sampling guarantee. A seed may improve repeatability, but model versions, backend kernels, tokenisation, prompts, tool results, and provider settings can still affect the result.

Should I use top-k or top-p?
Top-k gives a fixed candidate count, which is useful when you want a hard limit. Top-p adapts to the model’s confidence and is often a better general-purpose control. Neither is universally better. Test on representative prompts, especially ambiguous ones where an aggressive filter may discard a useful continuation.

What happens if I set both?
The system applies both restrictions according to its decoding implementation, leaving a narrower final set of candidates. Because filter order can differ, do not infer exact behaviour from parameter names alone. Read the provider’s documentation and test a saved set of logits or representative prompts.

Say this in the interview

“Temperature reshapes the probabilities, top-k keeps a fixed number of the best candidates, and top-p keeps the smallest candidate set containing a chosen share of probability mass; they control randomness and candidate filtering, not factuality or structural validity.”