Sampling — temperature, top-k, top-p

The model outputs a probability distribution over ~100k tokens. Sampling is the rule for picking one. This is the dial you tune to trade off between “predictable and correct” and “creative and varied”.

Greedy: pick the top

The simplest strategy: pick whichever token has the highest probability.

probs = {"Paris": 0.62, "the": 0.18, "located": 0.12, "a": 0.08}

greedy_pick = max(probs, key=probs.get)   # always the argmax
print("greedy picks:", greedy_pick)

greedy picks: Paris

Greedy is deterministic given the same probabilities. It’s fast and boring — exactly what you want for classification, extraction, and structured outputs.

Temperature: the creativity dial

Temperature T rescales the logits (pre-softmax scores) by dividing them by T. T = 1.0 is the model’s “natural” distribution. T < 1 makes the distribution sharper (peaks get peakier). T > 1 makes it flatter (rare tokens get more chance). The intuition: dividing by a small T amplifies the gap between high and low logits before softmax, so the already-highest score dominates even more; dividing by a large T compresses the gap, so scores converge and all tokens become nearly equally likely.

import math

logits = {"Paris": 4.0, "the": 2.5, "located": 2.0, "a": 1.0}

def softmax_with_temperature(logits, T):
    if T <= 0:                                  # T=0 collapses to argmax
        top = max(logits, key=logits.get)
        return {k: (1.0 if k == top else 0.0) for k in logits}
    scaled = {k: v / T for k, v in logits.items()}
    m = max(scaled.values())
    exps = {k: math.exp(v - m) for k, v in scaled.items()}
    Z = sum(exps.values())
    return {k: v / Z for k, v in exps.items()}

for T in [0.1, 0.7, 1.0, 2.0]:
    p = softmax_with_temperature(logits, T)
    print(f"T={T}:", {k: round(v, 3) for k, v in p.items()})

T=0.1: {'Paris': 1.0, 'the': 0.0, 'located': 0.0, 'a': 0.0}
T=0.7: {'Paris': 0.841, 'the': 0.099, 'located': 0.048, 'a': 0.012}
T=1.0: {'Paris': 0.71, 'the': 0.158, 'located': 0.096, 'a': 0.035}
T=2.0: {'Paris': 0.485, 'the': 0.229, 'located': 0.178, 'a': 0.108}

Watch the distribution flatten as T climbs. At T=0.1 Paris is effectively certain; by T=2.0 even “a” carries real weight and the tail tokens have a genuine chance. The same three distributions, drawn as bars:

Top-k: keep the best k

Top-k truncates the distribution to the k highest-probability tokens before sampling. Everything else gets zero probability. Cuts off the long tail of unlikely garbage.

# Pseudocode for top-k
sorted_tokens = sorted(probs.items(), key=lambda x: -x[1])
top_k = dict(sorted_tokens[:k])
# Renormalize so they sum to 1
total = sum(top_k.values())
top_k = {t: p / total for t, p in top_k.items()}

Typical k is 40-50. Modern APIs use top-p more than top-k, but both exist.

Top-p (nucleus): keep tokens up to cumulative p

This is the most useful one to understand. Top-p sorts tokens by probability, then keeps only the smallest set whose cumulative probability exceeds p (typically 0.9 or 0.95). The “nucleus” of the distribution.

def top_p_sample(probs, p_threshold=0.9):
    sorted_items = sorted(probs.items(), key=lambda x: -x[1])
    cumulative = 0.0
    nucleus = []
    for tok, prob in sorted_items:
        cumulative += prob
        nucleus.append((tok, prob))
        if cumulative >= p_threshold:       # stop once we cross p
            break
    total = sum(p for _, p in nucleus)      # renormalise the nucleus
    return {t: p / total for t, p in nucleus}

probs = {
    "Paris": 0.45, "Lyon": 0.20, "Marseille": 0.15,
    "Nice": 0.08, "Lille": 0.05, "Toulouse": 0.04,
    "Bordeaux": 0.02, "Nantes": 0.01,
}

print("nucleus (p=0.9):")
for tok, p in top_p_sample(probs, 0.9).items():
    print(f"  {tok}: {p:.3f}")

nucleus (p=0.9):
  Paris: 0.484
  Lyon: 0.215
  Marseille: 0.161
  Nice: 0.086
  Lille: 0.054

The cumulative probability crossed 0.9 at Lille (0.45 + 0.20 + 0.15 + 0.08 + 0.05 = 0.93), so the three rare tail tokens — Toulouse, Bordeaux, Nantes — were dropped entirely, and the surviving five were renormalised to sum to 1.

Why top-p over top-k? Top-p adapts. When the model is confident (“the capital of France is __”), the nucleus might be just 1-2 tokens. When it’s open-ended (“once upon a time __”), the nucleus might be 50. Top-k forces the same k regardless of how peaked the distribution is.

A crucial reminder the three together make clear: the knobs do not pick tokens, they reshape the distribution you then draw from. Temperature always applies; top-k and top-p are filters that trim the candidate set first. Set the shape, then sample.

What to use when

This is a cheat-sheet, not gospel. Most APIs default to reasonable values; you only need to override for specific reasons.

A subtle gotcha

If you’re A/B testing prompts, run each prompt multiple times unless temperature is 0. A single sample at T=0.7 is noisy. You can convince yourself prompt B is better when it just got lucky once. Average over at least 5-10 samples per prompt, or run a real eval.

In one breath

• Greedy picks the argmax — deterministic, ideal for classification, extraction, and JSON.
• Temperature rescales logits before softmax: T < 1 sharpens, T > 1 flattens.
• Top-k keeps the k best tokens; top-p keeps the smallest set summing to p — and adapts to how peaked the step is.
• temperature=0 is mostly but not bit-reproducible (GPU float order + ties); add a seed if you need more.
• Tune either temperature or top_p, not both; and run >1 sample when T > 0 before trusting an A/B.

Quick check

Next

The next lesson covers structured outputs — how to actually get the model to emit valid JSON every time, using schemas instead of hoping.