Glossary/T/temperature
Model behaviour Also called: sampling temperature, softmax temperature

What is temperature?

Temperature is a sampling parameter that rescales the model's next-token probabilities before one is drawn. A value near zero flattens the choice toward whichever token already scored highest, so the output looks nearly deterministic. A value near one leaves the distribution as trained, which admits more of the tail. It is a knob on how the sampler reads the logits, not a knob on how the model thinks.

Written by Derek Wonk · last updated August 2026

Ask ChatGPT Ask Claude View as Markdown

What temperature actually does to the logits

After a forward pass the model produces a vector of logits, one score per token in the vocabulary. The sampler divides those logits by the temperature and then applies softmax to turn them into a probability distribution. Dividing by a small number magnifies the gap between the top logit and everything else, so the softmax concentrates on the leader and the sampler almost always draws it. Dividing by a larger number shrinks the gaps, so lower-ranked tokens keep enough probability to be picked.

The value the API accepts is not the whole story. Providers pin the parameter to a range that fits their model: OpenAI accepts 0 to 2, Anthropic accepts 0.0 to 1.0, and the two are not directly comparable because the underlying scaling is applied to different logit distributions. A temperature of 0.7 on one provider is not the same operating point as 0.7 on another, which is why prompts ported between models sometimes look tighter or looser than they used to.

Why is temperature 0 not fully deterministic?

Temperature of zero degenerates the sampler into an argmax: always take the highest-scoring token. That removes randomness from the choice step, but a production request still passes through a stack that does not guarantee bitwise repeatability. Batched GPU kernels reduce floating-point sums in nondeterministic order, mixture-of-experts routing depends on the batch a request lands in, and providers roll silent point updates to a model version. Anthropic states this outright in the Messages reference: even at temperature 0.0 the results will not be fully deterministic.

If reproducibility matters, do not lean on temperature alone. Pin the model version, use a seed where the provider supports one, and treat identical output across runs as a nice-to-have rather than a contract.

Temperature is not creativity, and it interacts with top-p

The marketing framing calls low temperature analytical and high temperature creative. The mechanism is narrower than that: the model does not know it is being sampled differently, and a bad idea at temperature 0.2 is still a bad idea at temperature 1.2, just phrased with more surprising word choices. If an output is wrong, raising temperature will not fix it; if an output is repetitive, the cause is often the prompt or the tokeniser rather than the sampler.

Temperature also composes with the truncation samplers. Top-p (nucleus sampling, from Holtzman et al.) keeps only the smallest set of tokens whose cumulative probability exceeds p, then renormalises. That truncation happens after the temperature rescale, so setting both high stacks their effects and produces the sort of runaway output the nucleus paper was written to explain. Most providers advise changing one of the two, not both.

Sensible defaults for real tasks

Reach for low temperature (0 to 0.3) when the shape of the answer is constrained: classification, extraction, structured output, tool arguments, code that has to compile. Reach for the trained default (around 0.7 to 1.0 depending on provider) when you want variety across runs, for example brainstorming, style rewrites or synthetic data. Values above one are rarely useful outside of research: the distribution flattens far enough that grammar and coherence start to slip.

For evals, hold temperature fixed and vary only the prompt or the model. A moving temperature between two runs makes the diff impossible to attribute, which is the most common reason a numbers-driven prompt experiment fails to draw a clear conclusion.

The same prompt at four temperatures

prompt: "Continue: The server returned a 500 because"

temperature = 0.0
  the database connection pool was exhausted.
  the database connection pool was exhausted.
  the database connection pool was exhausted.

temperature = 0.3
  the database connection pool was exhausted.
  the database connection pool had run out of available connections.
  the database connection pool was exhausted and the request timed out.

temperature = 0.7
  the request timed out waiting for a Redis reply.
  an unhandled KeyError propagated out of the auth middleware.
  the JSON body exceeded the configured 1 MB limit.

temperature = 1.3
  the shard hosting the tenant was mid-failover; retries hit the old primary.
  a background migration held an ACCESS EXCLUSIVE lock on the users table.
  the sidecar was OOM-killed and dropped the trailing chunk of the response.

Low temperature repeats the argmax. Raising it broadens the plausible completions without making any single answer more correct.

Common questions

Temperature: frequently asked

Is temperature the same as creativity?

No. It is a rescale on next-token probabilities, applied by the sampler after the model has already computed its scores. Higher values let lower-ranked tokens win more often, which reads as more varied surface language. It does not change what the model knows or how it reasons, so a wrong answer at low temperature stays wrong at high temperature.

Should I change temperature or top-p?

One at a time. Both narrow or widen the pool the sampler draws from, and stacking them makes it hard to attribute an output change to either. A common practical stance is to leave top-p at its default and vary temperature, or vice versa, and only reach for both once you understand what each does on its own.

Why did temperature 0 still give me different outputs?

Temperature 0 removes randomness from the sampling step but not from the rest of the stack. Batched floating point reductions, expert routing in mixture-of-experts models, and silent provider updates to a model version all introduce variation across otherwise identical requests. Pin the model version and use a seed if the provider offers one; treat exact repeatability as best-effort.

What temperature should I use for structured output or tool calls?

Something low, typically 0 to 0.2. Structured output has one right shape and many wrong shapes, so you want the sampler to stay near the argmax. Tool calling is the same argument: the arguments are a schema-constrained object, and randomness in field values or names shows up as parse failures downstream.

Sources

Where these facts come from

token model reasoning prompt tool calling