Sampling — temperature, top-k, top-p

Your support bot answers the same customer question three times. The first answer is concise. The second invents a refund policy. The third is oddly poetic about shipping labels.

The model did not suddenly acquire a personality. At every step, it had several possible next tokens and chose among them. The rule used to make that choice is sampling.

A language model produces scores for every token in its vocabulary. A token is a small piece of text: sometimes a whole word such as Paris, sometimes a fragment such as ing, or punctuation. Sampling turns those scores into a choice, repeated for every token in the response.

Sampling does not choose an entire answer at once. It chooses one next token, then calculates a new distribution after seeing that token.

The mental model: scores, probabilities, and one draw

Suppose the prompt ends with:

The capital of France is

For the next token, imagine:

probs = {"Paris": 0.62, "the": 0.18, "located": 0.12, "a": 0.08}

These probabilities describe uncertainty at this one position. They do not mean the whole answer has a 62 percent chance of being correct.

Greedy decoding always picks the most probable token, or argmax: here, Paris. Sampling instead draws from the distribution. A draw in the first 62 percent produces Paris; a draw between 62 and 80 percent produces the.

The model then sees the new ending, The capital of France is the, and calculates a fresh distribution. The decision is local, but the response is a chain of local decisions. The probability of a complete sequence is the product of its token probabilities, so many individually plausible choices can still form an unlikely paragraph.

Models usually start with logits, raw scores rather than probabilities. Softmax converts them:

probability(token i) = exp(logit i) / sum(exp(all logits))

Temperature changes logits before softmax. Top-k and top-p remove candidates after probabilities have been shaped. A final random draw chooses from what remains.

Greedy decoding: always take the peak

Greedy decoding is deterministic if the model produces exactly the same scores. It is cheap and predictable, making it a sensible starting point for classification, extraction, routing, and structured output.

It is not automatically better. Always taking the peak can produce repetitive wording in long, open-ended text. And if the model’s distribution is wrong, greedy decoding chooses the wrong peak consistently.

Temperature: reshape the whole distribution

Temperature is a positive number that rescales logits before softmax. If a logit is z, temperature uses z / T.

• T = 1.0 leaves the distribution at its natural scale.
• T < 1 sharpens it, making high-probability tokens dominate.
• T > 1 flattens it, giving less likely tokens more probability.
• Many providers interpret T = 0 as greedy or near-greedy decoding, but its semantics are provider- and model-specific. The system does not literally divide by zero.

Here is the arithmetic with four simplified logits:

import math

logits = {"Paris": 4.0, "the": 2.5, "located": 2.0, "a": 1.0}

def softmax_with_temperature(logits, T):
    if T < 0:
        raise ValueError("temperature must be non-negative")
    if T == 0:                                  # special-case greedy decoding
        top = max(logits, key=logits.get)
        return {k: (1.0 if k == top else 0.0) for k in logits}
    scaled = {k: v / T for k, v in logits.items()}
    m = max(scaled.values())                    # numerical stability
    exps = {k: math.exp(v - m) for k, v in scaled.items()}
    Z = sum(exps.values())
    return {k: v / Z for k, v in exps.items()}

for T in [0.1, 0.7, 1.0, 2.0]:
    p = softmax_with_temperature(logits, T)
    print(f"T={T}:", {k: round(v, 3) for k, v in p.items()})

T=0.1: {'Paris': 1.0, 'the': 0.0, 'located': 0.0, 'a': 0.0}
T=0.7: {'Paris': 0.841, 'the': 0.099, 'located': 0.048, 'a': 0.012}
T=1.0: {'Paris': 0.71, 'the': 0.158, 'located': 0.096, 'a': 0.035}
T=2.0: {'Paris': 0.485, 'the': 0.229, 'located': 0.178, 'a': 0.108}

At T=1.0, the logit gap between Paris and the is 1.5. At T=0.5, it becomes 3.0, strongly favoring Paris; at T=2.0, it becomes 0.75, allowing the other candidates to catch up.

Temperature does not add a “creative token.” At T=2.0, a becomes more likely because flattening makes the previously unlikely continuation less unlikely.

High temperature exposes more of the low-probability tail, where malformed syntax, repetition, contradictions, and accidental role changes are more likely.

A provider may reject or ignore a sampling parameter for a particular model or endpoint. If changing temperature produces identical outputs, check the API documentation and actual request configuration.

Top-k: keep a fixed number of candidates

Top-k sampling sorts candidates by probability and keeps only the k most likely. All other tokens receive zero probability; the survivors are then renormalised.

With Paris: 0.62, the: 0.18, located: 0.12, and a: 0.08, k=2 keeps Paris and the. Their probabilities become approximately 0.775 and 0.225, because each is divided by the surviving mass, 0.80. k=1 is effectively greedy decoding.

Top-k’s weakness is its fixed count. When the model is confident, a large k may include absurd options; when it is uncertain, a small k may be too restrictive.

Top-p, or nucleus sampling: keep a variable mass

Top-p sampling keeps the smallest set of highest-probability tokens whose cumulative probability reaches a threshold p.

For this distribution:

probs = {
    "Paris": 0.45, "Lyon": 0.20, "Marseille": 0.15,
    "Nice": 0.08, "Lille": 0.05, "Toulouse": 0.04,
    "Bordeaux": 0.02, "Nantes": 0.01,
}

For p=0.9, accumulate in order:

Paris                         0.45
Paris + Lyon                  0.65
+ Marseille                   0.80
+ Nice                        0.88
+ Lille                       0.93

The nucleus therefore contains five tokens. Discard the rest and divide each survivor by 0.93:

Paris 0.484, Lyon 0.215, Marseille 0.161, Nice 0.086, Lille 0.054

Top-p adapts to uncertainty. At a confident step, one token may carry 94 percent of the probability, producing a tiny nucleus. At an open-ended step, the nucleus may contain dozens of reasonable continuations.

A high p, such as 0.99, leaves much of the tail available. A low p can remove useful uncommon words, names, or code tokens.

The conceptual pipeline is:

1. Rescale logits with temperature.
2. Convert them to probabilities.
3. Apply top-k or top-p filtering.
4. Renormalise the survivors.
5. Draw one token.

Implementations can arrange these processors differently, so read the provider’s documentation before transferring settings between libraries.

Combining the controls

Temperature changes probability mass. Top-k and top-p delete candidates; they are not three independent “creativity sliders.”

If both filters are enabled, a token must survive both. Top-k might leave 40 candidates, while top-p reduces those to 12. The exact order is implementation-dependent, but the result is narrower than either filter alone.

Tune one control at a time:

• Keep top_p=1.0 while tuning temperature, or
• Keep temperature=1.0 while tuning top-p.

top_p=1.0 means no nucleus truncation in the usual definition. Neutral settings can still be stochastic because sampling itself remains random.

Choosing settings in practice

There is no universal production default. A model’s training, instruction tuning, endpoint, and output constraints matter more than a copied magic number.

These are starting points, not promises. Measure the task you actually care about: extraction accuracy, JSON validity, factual support, code-test pass rate, or human preference.

Failure modes you can recognise

The answer repeats itself. Very low temperature or an over-restrictive top-p can narrow the available continuations. Raise the temperature slightly, widen the nucleus, shorten the output, or improve the prompt. Do not jump straight to a high temperature; that trades repetition for noise.

The answer becomes syntactically or factually wild. Broken JSON, invented citations, abrupt topic changes, and malformed code suggest too much low-probability mass or an underspecified prompt. Lower temperature or top-p, add constraints, and validate. No sampler can provide missing knowledge.

Changing temperature appears to do nothing. Check the payload and endpoint. The parameter may be unsupported, ignored, clamped, or overridden. Compare several outputs or distributions, not one response.

temperature=0 gives different answers. Check for a changing model snapshot, system prompt, tool result, retrieval order, tokenizer, or server routing. With fixed inputs, near-tied logits and numerical differences can still change a token. A seed helps supported stochastic decoders but is not a universal reproducibility guarantee.

A small but important distinction

Sampling controls output variation, not model capability. If the model assigns 70 percent probability to a false answer and 30 percent to the truth, lowering temperature usually makes the false answer more consistent. The sampler does not fact-check.

Use LLM evals to measure whether a setting improves your task. Use caching when you need repeated identical results or lower cost.

Next

The next lesson covers structured outputs: how schemas and constrained decoding make machine-readable responses dependable instead of merely hopeful.

Quick check