execution

Decoding and Sampling

Logits express a preference. The decoder decides how that preference becomes a token.

Published 2026-05-29 · Updated 2026-10-06
On this page

Inference-time decoding controls do not retrain the model. They modify the distribution or decision used to emit tokens. Temperature changes relative preference. Truncation changes eligibility. Guidance changes the scores themselves. A watermark creates a keyed statistical correlation in token selection.

These mechanisms solve different problems. They should not be treated as interchangeable “creativity settings.” Speculative decoding then accelerates a specified target sampler; it must account for the actual probabilities after these transformations.

The model is not the sampler

A vocabulary head produces logits z₁…zᵥ. Softmax converts them to positive probabilities. The next-token sampler can transform those scores before selecting an outcome.

pi=exp⁡(zi−m)∑jexp⁡(zj−m),m=max⁡jzjp_i=\frac{\exp(z_i-m)}{\sum_j\exp(z_j-m)},\qquad m=\max_jz_j
Subtracting the maximum leaves softmax unchanged and avoids overflow. Masks can set banned-token logits to −∞; an all-banned row needs an explicit error policy.

Greedy decoding chooses a maximum-logit token. Sampling chooses according to a distribution. Deterministic tie-breaking, seed management and floating-point kernels affect reproducibility. An identical RNG seed does not guarantee identical text across changed implementations.

Temperature changes relative preference

pi(T)=softmax⁡(z/T)i,pi(T)pj(T)=exp⁡ ⁣(zi−zjT)p_i(T)=\operatorname{softmax}(z/T)_i,\qquad\frac{p_i(T)}{p_j(T)}=\exp\!\left(\frac{z_i-z_j}T\right)
T must be positive for this formula. Temperature zero is usually implemented as a separate greedy branch.

T below one amplifies logit differences. T above one compresses them. Finite positive temperature preserves ranking and support before numerical rounding. It changes how much probability the leading tokens receive.

For logits 10 and 8, the ratio is exp(4) at T=0.5 and exp(1) at T=2. Lowering T does not tell the model which answer is true. It can make a confident error more likely.

As T approaches zero, probability concentrates on maximizers. If several logits tie, the mathematical limit shares mass across them. A deterministic greedy implementation may choose only one. State the tie policy.

Top-k and nucleus sampling

Top-k keeps a fixed number of highest-scoring tokens, zeros the rest and renormalizes. It changes support rather than only sharpness. k=1 gives greedy selection under a stated tie rule.

pi′={pi∑j∈Skpji∈Sk0i∉Skp'_i=\begin{cases}\displaystyle\frac{p_i}{\sum_{j\in S_k}p_j}&i\in S_k\\0&i\notin S_k\end{cases}
Sₖ is the chosen top-k set. If a threshold implementation retains ties at the boundary, it may keep more than k tokens.

For example, let the=0.30, a=0.20 and this=0.12 be the leading three token probabilities. Their total is 0.62, so top-3 probabilities become about 0.484, 0.323 and 0.194. These three values do not describe the complete vocabulary; assume every remaining token is below the cutoff.

Top-p, or nucleus sampling, sorts probabilities and keeps the smallest prefix whose cumulative mass reaches the threshold. Include the token that crosses the threshold. A confident distribution can keep one token; a broad one can keep many.

Transformation order matters. Temperature changes which nucleus meets a given probability mass. Combining top-k and top-p can mean applying nucleus to the already-renormalized top-k distribution, as in the example below. Other engines use different orders.

Same logits. Different sampler.

Four synthetic logits: [4, 2, 1, 0]. Order: temperature → top-k → nucleus → renormalize.

A0.831
B0.112
C0.041
D0.015
Entropy 0.859 bitsEligible tokens 4

Temperature does not add knowledge or verify factual correctness. Truncation creates exact zeros.

Common starting ranges—roughly 0–0.3 for focused tasks, 0.5–0.8 for general chat, 0.9–1.2 for creative text, and top-k around 10 or 40–50—are only starting heuristics. They are not portable recommendations for every model. Validate quality, failure rates and diversity on your workload.

LoopCD: extrapolate recurrent refinement

A looped transformer processes a prefix repeatedly with shared weights. The paper evaluates Ouro, Huginn, Parcae and Looped-Qwen3. An earlier completed loop and the final loop give two states for the same next-token decision. LoopCD extrapolates from the reference toward the final state and beyond.

Logit guidance:z′=zfinal+w(zfinal−zref)Hidden guidance:h′=hfinal+w(hfinal−href)\begin{aligned}\text{Logit guidance:}\quad z'&=z_{\rm final}+w(z_{\rm final}-z_{\rm ref})\\\text{Hidden guidance:}\quad h'&=h_{\rm final}+w(h_{\rm final}-h_{\rm ref})\end{aligned}
Decode h′ through the normal final normalization/coda/output head. Logit guidance decodes both states, so it needs an extra output pass.

Continue a change in preference

Reference logits [2,1]. Final logits [2.1,2]. Refinement increased B more than A.

Guided A 2.15Guided B 2.50Greedy choice B

This illustrates a ranking change, not evidence that B is correct.

A linear output map makes hidden and logit extrapolation commute. Actual output paths include normalization and sometimes nonlinear coda layers. The two variants can therefore change rankings differently.

w=wmax⁡[1−(ptop1−ptop2)]w=w_{\max}\left[1-(p_{\rm top1}-p_{\rm top2})\right]
Probabilities come from the unguided final logits; w_max≥0.

The adaptive logit rule uses the gap between the two leading probabilities. A small probability margin receives stronger guidance. Applying this exact rule to hidden guidance can require an extra output pass to obtain those probabilities, losing its principal cost advantage.

The source reports Ouro-2.6B-Thinking AIME 2024 pass@1 improving from 61.88% to 73.33% with logit guidance, and Huginn HumanEval from 22.56% to 31.71% with hidden guidance. Half-depth experiments report 22.5–48.2% fewer forward FLOPs on a multiple-choice suite while recovering baseline quality. The arithmetic estimate uses a 512-token prefill; it is not a measured serving speedup.

Choose strength on held-out data, then lock it before final evaluation. Compare unguided full-depth, guided full-depth and shorter guided runs. Keep checkpoint, prompt and sampler fixed. Start with w=0 to recover the baseline. Capture complete loop boundaries, align the coda interface, and preserve reference/final tensors without in-place edits or buffer reuse. Huginn’s noisy initialization needs a later reference after burn-in. A second coda branch must not overwrite the final branch’s KV state.

Excessive guidance amplifies errors. Measure generated token counts as well as loop counts: longer reasoning can erase the savings. Track peak memory, prefill time, time per generated token and throughput at realistic batches. Inspect repeated-token failures and temperature sensitivity. Transferring this update to a recurrent vision backbone is a hypothesis, not an established result.

LoopCD paper ↗ · Ouro reference →

Watermark the token choices

A text watermark is not metadata, hidden Unicode or a writing style. It is a deliberate statistical relationship between emitted tokens and a secret key. Copying the words preserves it; replacing the words can remove it.

SynthID-Text adds keyed sampling after ordinary distribution transformations. It requires no extra output tokens or model retraining. A detector reconstructs local keyed scores from a passage without needing the prompt or rerunning the LLM.

logits→temperature / truncation→keyed sampling→xtrt=hash⁡(xt−H:t−1,key),H=4\begin{aligned}\text{logits}&\to\text{temperature / truncation}\to\text{keyed sampling}\to x_t\\r_t&=\operatorname{hash}(x_{t-H:t-1},\text{key}),\qquad H=4\end{aligned}
The main paper configuration uses a four-token context. The same token can receive unrelated scores after different contexts.

This differs from a post-hoc AI detector. A post-hoc detector infers origin from naturally occurring patterns. A watermark deliberately creates a testable correlation.

Tournament generation and detection

Conceptually, m binary tournament layers begin with 2m independent samples from the model distribution. Candidate tokens can repeat. In each layer, pair candidates and select the higher keyed score under that layer’s function. Random tie-breaking handles identical scores.

This is the conceptual construction, not a requirement to materialize exponentially many candidates in production. The paper provides more efficient vectorized sampling.

S(x1:T)=1mT′∑t∈I∑ℓ=1mgℓ(xt,rt)S(x_{1:T})=\frac1{mT'}\sum_{t\in\mathcal I}\sum_{\ell=1}^m g_\ell(x_t,r_t)
Illustrative mean detector score: m≥1 layers, usable positions I, T′=|I|>0. This is not a calibrated probability of authorship.

The detector uses the expected tokenizer and key, reconstructs recent contexts, recomputes emitted-token scores and aggregates usable positions. The mean score above averages keyed values over usable positions and layers. Unwatermarked text supplies a random-score baseline; tournament winners tend to score higher. Weighted, frequentist and learned Bayesian scores can use the evidence differently.

A secret ordinary RNG seed alone does not provide this detector. Replaying an ordinary sampler would require the original prompt and model probabilities. Tournament scores can be reconstructed from the observed tokens themselves.

What non-distortion actually means

The two-competitor tournament has a single-token distribution-preservation guarantee averaged over the random keyed seed. This is not equality conditional on a fixed known seed or key. A particular keyed ranking deliberately favors some candidates.

Sequence-level guarantees are stronger than a marginal token guarantee. Repeated-context masking prevents repeatedly reusing the same random ranking in a way that accumulates distortion. The paper’s configurations distinguish single-sequence and multiple-sequence preservation.

More than two competitors per match can strengthen detection while distorting token probabilities. Stronger preservation can reduce detectability or diversity advantages. “Non-distortionary” therefore needs its averaging conditions and configuration, not an unqualified claim of identical text.

Entropy, editing and evidence

A nearly deterministic next-token distribution offers almost no watermark choice. Most tournament candidates are the same token. A broad distribution offers alternatives, so the key can influence which plausible token wins.

Long unchanged prose often supplies more evidence than short answers, rigid factual strings or low-entropy code. Text length alone is insufficient: many repeated or deterministic positions add little useful signal.

Changing one token replaces its score and changes the seeds for following positions in the context window. Minor edits can leave enough evidence. Paraphrasing, translation, heavy rewriting and fragment mixing weaken it. Semantic preservation is not token preservation.

A detection threshold trades false positives against missed watermarks. Calibration and abstention matter. A positive result supports consistency with a supported keyed generation process; it does not prove that the model invented the ideas or wrote every word.

A negative result is not proof of human authorship. The scheme does not identify a particular user, account or conversation. Keep provider attribution separate from personal attribution.

The key is a security boundary. Key exposure can enable forgery or scrubbing attempts. Detection APIs, rate limits, key rotation and key management can help, but watermarks do not solve universal AI detection. Provider participation is voluntary; open-weight deployments need not include compatible watermarking. Signed provenance, platform records and contextual evidence remain complementary.

Dated deployment context

Anthropic’s August 2026 announcement describes a SynthID-based text watermark rollout for models released on or after August 2, with earlier-model support described separately. This is a dated announcement, not proof that every historical Claude response contains a detectable watermark.

File provenance uses a different mechanism, such as signed C2PA metadata. A statistical token watermark and signed file credentials should not be treated as interchangeable.

Original Anthropic announcement ↗ gives the dated deployment details. Verify current model support and detection availability before relying on a specific service.

Sampler implementation contract

  1. Specify logits, dtype, temperature and transformation order.
  2. Specify banned-token, repetition and EOS policies.
  3. Define top-k ties, the nucleus crossing token and all-masked behavior.
  4. Normalize the final support before sampling.
  5. Record the actual proposal probabilities when speculative decoding needs them.
  6. Distinguish reproducibility from distributional equivalence.
  7. Evaluate accuracy, output length, repetition, diversity and throughput under the same settings.

For a watermark, also specify tokenizer, context length, repetition masking, key configuration and detector calibration. A toy tournament diagram is not a cryptographic implementation.

Speculative Decoding → preserves a specified target distribution; LoopCD deliberately changes one. This distinction prevents calling every inference-time improvement “lossless.”

Sources and connected notes

Nucleus sampling ↗ · SynthID-Text paper ↗ · DeepMind explanation ↗

Speculative Decoding →Information Theory →Recurrent Models →Reasoning Compute →Serving →Performance →