Topic: Learning notes

Temperature and Top P: How a Model Selects the Next Token

Practical Prompt Engineering notes: use a five-token probability example to understand temperature, nucleus sampling, Top P 0.5, and the limits of consistent output.

Following the introduction to prompt engineering, this note continues Practical Prompt Engineering on Frontend Masters with Temperature and Top P. Both affect generation, but they adjust different things: temperature changes how concentrated the probability distribution is; Top P uses cumulative probability to decide which candidate tokens to retain.

The course questions brought several distinctions into focus: probability versus candidate count, consistent output versus correct content, and sampling concepts versus actual API support. This article uses an invented distribution with checked calculations. It does not call a model to compare generation quality.

Temperature changes how concentrated the distribution is

Suppose a model is selecting its next token from five candidates, A through E. The letters are labels. The table describes one generation step, not five complete responses.

The baseline is temperature = 1, with no Top P or other filtering applied:

Standard temperature scaling divides the model’s logits by T before applying softmax. Given the probabilities p_i at T = 1, the same transformation can be written as:

q_i = p_i^(1/T) / sum_j p_j^(1/T), T > 0

Lowering T gives higher-probability candidates a stronger advantage. Raising T narrows the probability gaps. A remains first in this example, but its chance of being sampled changes. Positive temperature scaling alone preserves the ranking; it does not move E straight to first place. TemperatureLogitsWarper documentation

Percentages are rounded, so columns may not sum to exactly 100%. Once a token is selected, the preceding text changes and the next distribution is computed again. This table is not a fixed menu for an entire response.

What do temperature 0 and 2 tell us?

The course uses 0 to illustrate selecting the most likely token. The formula above only applies to positive T. An API supporting temperature = 0 typically handles it separately as highest-probability selection. In Transformers, greedy decoding can also be selected explicitly rather than substituting zero into the scaling formula. Generation configuration

Reducing sampling variation does not establish that a service will return identical text on every request. Google’s generation parameter documentation still allows for variation at temperature 0. Checking reproducibility requires keeping the model version, complete input, settings, and observed results.

At temperature = 2, the example distribution becomes flatter, but the five tokens do not each have a 20% chance. “Two means completely random and always unreadable” is therefore not a general rule. Higher temperatures can reduce coherence; the extent depends on the model, input, and other sampling settings.

Two is not a universal API maximum either. Before using a service, check the selected model’s permitted range and whether adjustment is supported at all.

Top P uses cumulative probability, not a percentage of candidates

Top P is also called nucleus sampling. It orders candidates by decreasing probability, keeps the smallest prefix whose cumulative probability reaches the threshold, and samples from that retained set. Both the GenerationConfig definition and the original nucleus sampling paper describe filtering in terms of probability mass.

Using the same five candidates at T = 1:

With top_p = 0.5, A’s 40% is insufficient. Adding B reaches 65%, so A and B are retained. That is two out of five candidates, not the top 50% of candidates by rank. The sum need not equal 50% exactly; B, which crosses the threshold, remains included.

The probabilities are then renormalized. A has a sampling probability of 40 / 65 ≈ 61.54%, and B has 25 / 65 ≈ 38.46%. Retaining two candidates does not give each an equal chance.

The same distribution also gives these results:

• top_p = 0.9: retain A, B, C, and D, totaling 95% of the original probability mass.
• top_p = 1: Top P itself removes no candidates; other settings may still restrict them.

As distributions change between steps, the number of retained candidates can change too. This distinguishes Top P from Top K, which retains a fixed number of candidates.

How do the two parameters work together?

If an implementation applies temperature scaling before Top P, the changed distribution can also change the retained candidate set. Changing both parameters at once makes it difficult to attribute a difference in the final text to either adjustment. The two tables here deliberately separate the calculations; the Top P example does not use the T = 0.5 column.

Consult the SDK and model documentation for the actual processing order, other filters, and constraints. Holding one parameter fixed makes comparisons easier to interpret.

Decide what to observe for the task

The course associates lower temperatures with factual questions, code generation, and extraction, and higher temperatures with creative work and brainstorming. I treat that as an introduction to the tradeoff, then test against the particular task and model:

A lower temperature can still produce the same mistake consistently. A higher temperature does not ensure valuable creativity. “Use 1.3 for creative work” cannot replace evaluating the actual task.

The course’s parameter settings also cannot be transferred directly to every newer model. At the time of consultation, the Claude Messages reference and Google generation parameter documentation both restrict custom sampling settings for some models. A setting may be rejected or ignored. Confirm support before comparing results.

Turn the concepts into a small, checkable experiment

I would start with an original extraction example to make success concrete:

Extract browser, route, and error from the report below. Output JSON only.
Use null for any missing field. Do not fill in missing information.

Bug report:
Using Firefox, I opened /settings and clicked Save. The error was
TypeError: Failed to fetch.

The expected result, written before running the experiment, is:

{
  "browser": "Firefox",
  "route": "/settings",
  "error": "TypeError: Failed to fetch"
}

This is a test target, not an actual model response. First record the model version, complete prompt, supported parameters, and baseline settings. Change either temperature or Top P in each round. As an initial exploration, repeat each configuration five times, recording whether the JSON parses, whether all three fields match, and whether the response invents information absent from the report.

Then add cases without a browser or without an error to check the handling of null. Five results support an initial observation; choosing settings for a production application requires more representative cases. This proposed model experiment has not been run.

The comparison would separate three questions: did the parameter take effect, how much did the output vary, and was the result correct? A single fluent answer is insufficient to decide whether a sampling configuration suits the task.

References and verification scope

• Sabrina Goldfarb, Practical Prompt Engineering slides: slide 5 on Temperature and Top P.
• Course companion site, Temperature, Top P, Tokens, and Context Windows: this article covers only Temperature and Top P.
• Hugging Face, TemperatureLogitsWarper and GenerationConfig: scaling, sampling, and candidate filtering.
• Holtzman et al., The Curious Case of Neural Text Degeneration: the original nucleus sampling paper.
• Google, Content generation parameters: parameter support and reproducibility limits.
• Anthropic, Create a Message: model restrictions on temperature and Top P.

Sources consulted on 2026-10-15. Both probability tables use an invented distribution and checked calculations. The course answers’ descriptions of “the top 50% of candidates,” “zero always produces identical results,” and “two always produces gibberish” are replaced with the more precise explanations above. No model quality benchmark or personal proficiency assessment was performed.