What is sampling?

What sampling actually does after the forward pass

A forward pass leaves a vector of logits, one score per token in the model's vocabulary. Softmax turns those scores into a probability distribution over the same vocabulary. The sampler reads that distribution and returns one token, which the harness appends to the sequence before the next forward pass. Every decoding strategy is a rule for how the sampler picks: always the top score, weighted by probability, from a truncated set of candidates, or across several sequences at once.

Nothing in the sampler changes what the model knows. Two runs of the same model on the same input see the same logits; the sampler is what makes the second run produce different words. When a completion looks wrong or repetitive, the choice was made after the model finished the forward pass, not during it.

Greedy, multinomial, and beam: what are the choices?

Greedy decoding takes the argmax at every step. It is the simplest sampler and the one every API falls back to at temperature 0. Multinomial (or ancestral) sampling draws with probabilities proportional to the softmax output, so a token with 0.4 probability wins about 40% of the time. Truncation samplers like top-k and top-p sit on top of multinomial sampling: they discard the tail of the distribution first, renormalise, then draw. Temperature rescales the logits before any of that happens.

Beam search keeps several partial sequences alive at once and expands them in parallel, scoring each by its joint probability. It is the dominant sampler for machine translation and summarisation, where one high-likelihood completion is the goal. Chat and code models rarely use it, because beam search tends to produce bland, repetitive text on open-ended prompts, which is the failure mode the Holtzman nucleus paper diagnosed.

Where the sampler sits in the provider stack

Chat completion APIs expose the sampler indirectly through a small set of parameters. OpenAI's Chat Completions accepts temperature, top_p, presence_penalty, frequency_penalty, and seed. Anthropic's Messages accepts temperature, top_p, and top_k. Google's Gemini API accepts temperature, topP, topK, and candidateCount. In every case the provider applies temperature first, then top-k if set, then top-p, then a random draw. That order matters because each stage reads the distribution the previous stage left behind.

Local runtimes let you assemble the pipeline yourself. Hugging Face Transformers calls it a LogitsProcessor list; llama.cpp calls it a sampler chain. Both let you insert extra stages such as repetition penalties, min-p, or grammar-constrained decoding for structured output. That flexibility is why open-source pipelines can hit output shapes that hosted APIs cannot.

Why does sampling matter for evals and reproducibility?

Evals compare completions across model versions or prompts, and the sampler is a hidden variable in every comparison. A regression between two runs at temperature 0.7 might be the prompt, the model, or the sampler landing on a lower-probability token that call. The usual fix is to hold sampler parameters fixed across an eval, pin the model version, and either run each prompt many times or set temperature to 0 and accept that the output is greedy rather than representative.

Even at temperature 0 the results are not bitwise deterministic. Batched GPU kernels reduce floating-point sums in a nondeterministic order, mixture-of-experts routing depends on which batch a request lands in, and providers roll silent updates to a model version. Anthropic states this in the Messages reference. Pin the model, use a seed where the provider supports one, and treat repeatability as best-effort.

prompt: "The compiler rejected the change because"

softmax over the top of the vocabulary:

  0.38  the
  0.22  it
  0.14  of
  0.09  a
  0.06  our
  0.04  no
  ...

greedy (argmax)              -> "the"
multinomial (T = 1.0)        -> "it"   (drawn with p = 0.22)
top-p (p = 0.9, T = 1.0)     -> nucleus = {the, it, of, a, our}
                                renormalised, drawn "of"
top-k (k = 2, T = 1.0)       -> pool = {the, it}, drawn "the"
beam search (beams = 3)      -> keeps three candidate sequences alive
                                and returns the highest-scoring one