What is top-p?
Top-p is a sampling method that truncates the model's next-token distribution to the smallest set of tokens whose probabilities add up to at least p, then renormalises the survivors and draws from that set. It is also called nucleus sampling, from the 2019 Holtzman paper that introduced it. Top-p caps how far into the tail the sampler is allowed to reach on any given step, without committing to a fixed cutoff like top-k does.
What top-p actually does to the distribution
After the model produces logits and softmax turns them into probabilities, top-p sorts the tokens by probability and walks down the list, keeping tokens until their cumulative mass first reaches or exceeds p. Everything below that line is discarded, the remaining probabilities are renormalised so they sum to one, and the sampler draws from that reduced set. The set is what the paper calls the nucleus, and its size is not fixed: on a peaked step it can be a handful of tokens, on a flat step it can be hundreds.
That dynamic size is the whole point of the method. A fixed cutoff, like top-k with k equals 40, throws away useful tokens on flat distributions and lets in noise on peaked ones. Top-p lets the truncation follow the model's own confidence, which is why Holtzman et al. proposed it as an antidote to the repetitive text that greedy decoding and high-temperature sampling both produce.
Top-p vs top-k vs temperature: what is the difference?
Temperature rescales the logits before softmax, changing the shape of the distribution the sampler sees. Top-k keeps a fixed number of highest-probability tokens and drops the rest. Top-p keeps a variable number of tokens chosen by cumulative probability. Only temperature changes what the numbers are; top-k and top-p only decide which numbers get to compete for the draw.
The three are usually applied in that order by the provider's sampler stack: temperature first, then top-k if set, then top-p, then a random draw. That order matters because top-p reads whatever distribution the earlier steps left behind. Setting a high temperature and a high p at the same time widens the pool on both axes, which is the runaway configuration the nucleus paper was written to explain.
Why do providers say pick one, not both?
OpenAI's Chat Completions reference and Anthropic's Messages reference both include the same guidance: alter temperature or top_p, not both. The advice is operational rather than mathematical. Both knobs move the same thing, the effective breadth of the sampled distribution, so changing them together makes it hard to tell which one caused an output to shift. When an eval regresses after a parameter change and both were moved, you cannot attribute the regression, which is the situation you were running the eval to avoid.
The concrete rule most teams settle on is to leave one at its default and vary the other. If you are already varying temperature across tasks, hold p at 1.0 (no truncation) or at the provider's default. If you have a reason to want a hard cap on the tail, for example to reduce rare tokenisation glitches in structured output, set p somewhere in the 0.9 to 0.95 range and leave temperature at the default for that model.
Sensible defaults and when to change p
A p of 1.0 disables the truncation and returns the sampler to the full distribution shaped by temperature alone. Values in the 0.9 to 0.95 range are the working defaults most providers ship: they clip the very long tail, where individual tokens carry vanishing probability, without noticeably narrowing the pool the model can draw from. Below about 0.5 the sampler starts behaving like a greedy decoder with jitter, and you will see the repetition patterns the nucleus paper documents.
Structured output and tool calls are the usual reason to lower p rather than temperature. If the schema has one correct field name and many nearly-correct ones, a slightly tighter nucleus removes the near-misses that show up as parse failures downstream. For free-form generation, prefer the default; if the output feels too samey, raise temperature before touching p, because that is the knob whose effect on style is the more intuitive one.
Where the nucleus lands at three values of p
prompt: "The build failed because"
softmax probabilities over the top of the vocabulary:
0.42 the
0.19 a
0.11 Docker
0.08 npm
0.05 our
0.04 Postgres
0.03 yarn
0.02 Bazel
...
0.00 parliament
0.00 hyacinth
p = 0.5 -> nucleus = {the, a} cum. 0.61, 2 tokens
p = 0.9 -> nucleus = {the, a, Docker, npm,
our, Postgres, yarn} cum. 0.92, 7 tokens
p = 1.0 -> nucleus = the whole vocabulary cum. 1.00, ~50k tokens
after truncation the surviving probabilities are renormalised
to sum to 1, then one token is drawn.The nucleus is dynamic: it collapses to two tokens on a peaked step and would expand to hundreds on a flatter one. That is what top-k cannot do.
Common questions
Top-p sampling: frequently asked
Is top-p the same as top-k?
No. Top-k keeps a fixed number of highest-probability tokens, regardless of how the probability is distributed among them. Top-p keeps a variable number, chosen so their cumulative probability first reaches p. On a peaked distribution top-p may keep two tokens where top-k with k of 40 would keep 40; on a flat distribution top-p may keep 200 tokens where top-k would still cap at 40. They can be combined, and providers usually apply top-k first.
Should I change top-p or temperature?
One at a time. Both widen or narrow the pool the sampler draws from, so moving both makes it hard to attribute any output change to either. OpenAI and Anthropic both say this in their reference docs. A practical stance is to leave top-p at the provider default and vary temperature across tasks, then reach for top-p only when you have a specific reason to clip the tail, such as tightening structured output.
What value of p is a safe default?
The values providers ship, which sit in the 0.9 to 0.95 range for most chat models, and 1.0 (no truncation) for evaluation runs where you want the raw distribution. Below about 0.5 the sampler starts to behave like greedy decoding with a little jitter, and the repetition failure modes the nucleus paper documents start to appear. Above 0.95 there is rarely any benefit, because the extra tokens carry negligible probability.
Does top-p make output more deterministic?
It reduces variance by shrinking the set of tokens the sampler can draw from, which is not the same as removing randomness. A run at temperature 0.7 with p of 0.5 will still vary between calls, just within a narrower band of surface language. For reproducibility, rely on temperature 0 plus a fixed model version and a seed where the provider offers one, not on a low p.
Sources
Where these facts come from
- The Curious Case of Neural Text Degeneration (Holtzman et al., 2019): the paper that introduced nucleus (top-p) sampling and diagnosed the failure modes it was designed to fix
- Anthropic docs: Messages API reference (top_p): documents the top_p parameter and the guidance to alter either temperature or top_p, not both
- Hugging Face Transformers: text generation strategies: GenerationConfig reference for top_p, top_k and temperature and the order the sampler applies them