The same prompt sometimes produces materially different answers in production even though temperature is set to zero. What would you investigate across sampling parameters, model versions, batching, seeds, tool results, chat templates, and provider infrastructure?

How to think about it

Temperature zero usually requests greedy decoding, not a universal promise of identical text. I would verify the actual request and rendered token sequence first, then bisect differences in sampling rules, model revision, seed and batch execution, tool outputs, chat template, and provider routing.

Why temperature zero is not a magic determinism switch

An LLM generates one token at a time. At each step it produces logits, which are unnormalised scores for possible next tokens. Greedy decoding chooses the token with the highest score, written as argmax(logits).

At a positive temperature, the system typically divides logits by temperature before converting them into probabilities. Lower temperature makes the distribution sharper. At zero, many implementations use a special greedy-decoding path because literal division by zero is not meaningful.

That still leaves several ways for the winning token to change.

Suppose the next-token scores are:

"Eligible"      10.0000000
"Not"            9.9999998

A small floating-point difference from a different GPU kernel, reduction order, quantisation scheme, or model revision can reverse that choice. Once one token changes, every later token is generated from a different context. One microscopic numerical difference can therefore become a completely different paragraph.

There is another important distinction: temperature zero may be the value in your application configuration, but not the value that reached the provider. Check the serialised request, not the environment variable. Confirm whether the provider accepts zero, converts it to greedy mode, rounds it, rejects it, or applies a default.

I would inspect all parameters that can affect token selection:

• top_p, or nucleus sampling, limits choices to the smallest set whose probabilities reach a cumulative threshold.
• top_k limits choices to the highest-scoring k tokens.
• Frequency and presence penalties alter token scores before selection.
• Logit bias or equivalent controls can directly raise or lower particular tokens.
• A seed controls random-number generation when randomness is used. It does not make different model weights or different floating-point computations equal.
• Maximum output length and stop sequences usually affect where generation ends, not which earlier token wins. They can still matter in multi-step tool loops or when an application retries after truncation.

If the implementation is genuinely greedy and the top token has a comfortable margin, top_p, top_k, and the seed should normally have no practical effect. If scores are tied or nearly tied, implementation details matter.

A concrete production example

Imagine a support assistant answering:

Can I get a refund for order 4815?

The assistant is instructed to call an order_lookup tool before answering. At 14:02, the tool returns:

{"order_id":4815,"status":"delivered","delivered_at":"2026-08-21T14:11:00Z"}

At 14:07, the same-looking request returns:

{"order_id":4815,"status":"delivered","delivered_at":"2026-08-21T14:18:00Z"}

The difference is only seven minutes, but the refund policy has a seven-day cutoff. If the assistant is close to that boundary, “eligible” and “not eligible” are both plausible results. Temperature zero cannot make changing evidence produce a fixed answer.

Now suppose the tool result is byte-for-byte identical. I would replay the exact conversation at concurrency one, then under the production concurrency level. If the first differing token appears only under load, batching or provider infrastructure becomes suspicious.

A useful first step is to fingerprint the exact context sent to the model. The fingerprint is not a substitute for retaining a secure replayable copy, but it quickly proves whether two requests really had the same messages and tool result.

import json
from hashlib import sha256

def fingerprint(messages, tool_result):
    payload = json.dumps(
        {"messages": messages, "tool_result": tool_result},
        sort_keys=True,
        ensure_ascii=False,
        separators=(",", ":"),
    )
    return sha256(payload.encode("utf-8")).hexdigest()

print(fingerprint(
    [{"role": "user", "content": "Can I get a refund for order 4815?"}],
    {"order_id": 4815, "status": "delivered"},
))

I would compare the first divergent token, not just the final answer. If the first token differs, inspect decoding and execution. If the model first calls a tool identically but the final answer differs, inspect the returned tool payload, tool-result formatting, and the second model call.

The investigation matrix

Area	What I would compare	Typical first clue
Sampling	Actual temperature, top_p, top_k, penalties, logit bias, stop settings	Divergence at the first generated token
Model	Immutable model revision, tokenizer, adapter or fine-tune, quantisation	All traffic changes after a deployment
Batching	Batch size, continuous batching, padding, cache path, speculative decoding	Differences appear only during load
Seeds	Whether a seed is supported, recorded, and consumed consistently	Same seed works in a single process but not across replicas
Tools	Arguments, result bytes, ordering, retries, timestamps, database state	Divergence begins after a tool call
Template	Rendered prompt, special tokens, system text, tool schema, truncation	“Same messages” have different token IDs
Infrastructure	Region, replica, accelerator, runtime, provider alias, fallback route	Different answers correlate with host or region

Model versions and provider routing

Never treat a friendly model name as an immutable model. Providers may move an alias to a new checkpoint, apply a rolling deployment, change a tokenizer, or route requests to different hardware. Fine-tuned adapters and safety layers can change too.

Record the provider’s model identifier and revision when available, along with region, request ID, deployment identifier, and any exposed model or runtime fingerprint. Also check for silent fallbacks during incidents. A request intended for one model may receive a smaller or older model when capacity is constrained.

The same applies to self-hosted serving. Compare container image, weights checksum, tokenizer files, quantisation method, compiler version, GPU type, and inference engine settings. A model served with different quantisation is not the same numerical program, even when the source weights have the same name.

Batching and seeds

Batching should not change the mathematical answer for independent requests. Real hardware does not perform exact real-number arithmetic, however. Matrix operations and reductions use finite-precision arithmetic, and changing the batch shape can change kernel selection and accumulation order. Since floating-point addition is not perfectly associative, a tiny score change can alter a close argmax.

Continuous batching, prefix caching, and speculative decoding deserve explicit testing. They may be perfectly correct, but they create additional execution paths. Run the same request:

1. alone, repeatedly;
2. in a fixed batch;
3. under production concurrency;
4. on each relevant replica or region.

A seed only helps when the same random process is being replayed. With true greedy decoding, there may be no random draw for the seed to control. With sampling, tie-breaking, or speculative decoding, random-number consumption can depend on rejected tokens, batch scheduling, or implementation details. “Same seed” is therefore a scoped reproducibility claim, not a cross-provider guarantee.

Tool results and chat templates

“Same prompt” often means the same user sentence, not the same model input. A chat template is the formatting rule that turns messages into the token sequence consumed by the model. It may add system text, role markers, beginning-of-sequence tokens, an assistant-generation marker, tool schemas, or whitespace.

Two clients can send identical message objects and produce different token IDs because one uses a newer template or includes a different tool schema. Log the rendered prompt or its secure equivalent, tokenizer version, token IDs, message order, and truncation decision. Check whether a middleware layer inserts a safety instruction or trims old history.

Tools are part of the prompt pipeline. Log the exact tool arguments and returned bytes, including ordering of JSON fields if the model sees raw JSON, retry messages, errors, current timestamps, retrieved documents, search ranking, and database snapshot. A tool that returns “no result” after a timeout is not equivalent to a tool that returns an empty array.

The senior-level nuance

Absolute determinism is often the wrong product requirement. A live assistant depends on live data, and forcing every component into a deterministic mode can reduce availability, freshness, or useful exploration. For a refund decision, deterministic business rules should enforce the cutoff; the LLM can explain the result. Do not ask a language model to be the database and the policy engine because its temperature is zero.

For reproducible tests, pin the model revision, tokenizer, template, sampling parameters, seed, tool fixtures, runtime, and serving path. For production, add semantic checks, structured output validation, idempotency, and audit logs. A provider may still offer only best-effort reproducibility across hardware or deployments.

The most useful failure symptom is often load correlation: requests are stable at low traffic and diverge during a busy period. That points toward batching, routing, fallback models, or a timing-sensitive tool before it points toward temperature.

What they’ll ask next

Can top_p matter when temperature is zero?
If the server truly switches to greedy decoding and the top token is unique, usually not. But APIs differ, and filtering may happen before selection. I would verify behaviour experimentally and inspect the effective request rather than relying on the parameter name.

Will setting a seed fix the problem?
Only within a controlled implementation that supports the seed and consumes randomness the same way. A seed cannot fix different tool results, chat templates, model revisions, hardware paths, or a changed batch schedule.

How would you make the system deterministic?
Pin every relevant input and execution component, replay fixed tool responses, disable unneeded speculative or dynamic paths, and test at the same concurrency. Even then, treat cross-version and cross-provider determinism as a best-effort property, and use rules plus validation for decisions that must be stable.

One line to say in the room: “Temperature zero gives me greedy selection, not a determinism guarantee, so I would find the first divergent token and compare the exact request, tool context, model revision, runtime path, and batch conditions around it.”