Temperature and Top-p Explained: A Visual AI Beginner’s Guide
Different possibilities.
Raju asks his AI helper for a name for his ginger tea. One time it suggests “Ginger Comfort.” Another time it suggests “The Ginger Pause.” Why can the same request produce different wording? Today, we explore temperature and top-p—two generation settings that influence how a language model chooses among possible continuations.
Raju is fictional. These AI-generated cartoons are visual metaphors; the robot represents software. Names and outputs are teaching examples, not measured results.
01. First, remember the next-token idea
A language model produces scores for possible next tokens based on the context. A decoding method turns those scores into a choice. After a token is added, the model considers the updated context and repeats. A token can be part of a word, a whole word, or punctuation.
Context
Raju’s ginger tea is…
Candidates
Possible next tokens have scores
Selection
Choose a token, then continue
02. What does sampling mean?
Imagine a weighted draw: more likely tokens get a larger share of the chances. Sampling chooses using those probabilities, so a less likely token can sometimes be selected. Greedy decoding instead chooses the highest-scoring available token at each step.
Greedy
Pick the highest-scoring token
Sampling
Draw according to probabilities
03. What is temperature in AI?
Temperature changes how concentrated the next-token probabilities are. Lower positive values sharpen the distribution toward favored tokens; higher values flatten it, giving alternatives more opportunity. Temperature 1 leaves the original softmax distribution unchanged before other filters. It is a mathematical setting, not the computer’s physical temperature.
Lower
More concentrated choices
T = 1
Unscaled reference distribution
Higher
More spread among alternatives
04. A small numerical example
Suppose only two tokens are possible: A has probability 80% and B has 20% at temperature 1. At temperature 0.5 their probabilities become about 94.1% and 5.9%. At temperature 2 they become about 66.7% and 33.3%. A remains favored, but the gap changes.
T = 0.5
A: 94.1% · B: 5.9%
T = 1
A: 80% · B: 20%
T = 2
A: 66.7% · B: 33.3%
Optional: the simple math behind the example
For positive temperature T, raise each original probability to the power 1/T, then divide each result by their sum. For T = 0.5: 0.8² ÷ (0.8² + 0.2²) ≈ 0.941. Equivalently, temperature divides model scores called logits before softmax converts them to probabilities. You can understand the lesson without memorizing this formula.
05. What is top-p, or nucleus sampling?
Top-p limits the candidate pool by cumulative probability. Rank tokens from most to least likely. Keep the smallest leading group whose probabilities reach or exceed p, then sample within that group using rescaled probabilities. The token that crosses the threshold stays in the pool.
Rank
Highest probability first
Accumulate
Stop when the total reaches p
Sample
Choose among retained tokens
06. Walk through top-p = 0.8
Consider four hypothetical tokens: A = 50%, B = 25%, C = 15%, and D = 10%. A alone reaches 50%; A + B reaches 75%, still below 80%. Adding C reaches 90%, so A, B, and C remain. D is excluded.
A · keep
50% total
B · keep
75% total
C · keep
90% total: threshold reached
D · exclude
Outside the retained pool
07. Temperature and top-p do different jobs
Temperature reshapes relative probabilities. Top-p removes the lower-probability tail beyond a cumulative cutoff. They can interact because changing probabilities can change which tokens fit inside the cutoff. Exact processing order and other filters depend on the implementation.
Temperature
Changes probability concentration
Top-p
Limits the cumulative candidate pool
Top-p = 1
No nucleus truncation by this setting
08. Zero temperature is not a truth switch
Some services interpret temperature 0 as greedy or near-greedy decoding; other libraries require a separate greedy option and a positive sampling temperature. Even when repeated answers are more stable, that does not establish correctness. Reproducibility can also depend on model version and serving details.
Consistency
Does wording repeat?
Correctness
Are the claims supported?
Reproducibility
Can the run be recreated?
09. Raju still needs to check the facts
Suppose an output describes his tea as “award-winning,” although he supplied no award information. Lowering the temperature does not verify that claim. Raju should compare the draft against confirmed facts and remove unsupported details, whatever settings were used.
Source
Confirmed ingredients and shop details
Draft
Check each factual claim
Decision
Verify, revise, or remove
10. Where can you change these settings?
Some developer playgrounds, APIs, and local model tools expose temperature and top-p. A normal chat interface may hide them. Available controls, accepted ranges, and recommended defaults vary by model. Writing “temperature = 0.2” in a message does not itself change an API parameter.
Prompt
The task, facts, and output requirements
Parameter
A supported setting in the tool or API
Documentation
Which settings this model accepts
11. Run a useful beginner experiment
Keep the model, prompt, input facts, and output limit fixed. Generate five responses with the documented defaults. If the model supports tuning, change only temperature to another supported value and generate five more. Compare variety, instruction-following, and unsupported claims. Leave top-p unchanged for this first experiment.
Baseline
5 responses with default settings
One change
5 responses with one supported change
Compare
Variety + compliance + factual support
Suggest three short menu names for ginger tea. Confirmed ingredients: black tea, milk, ginger, and sugar. Use simple English. Do not invent awards, prices, ingredients, or health benefits. Return only the three names.
No settings in your chat app? You can still practice writing clear prompts and checking answers. Asking for several alternatives is useful, but it is not the same experiment as changing a sampling parameter.
12. Choose settings by testing the task
For a menu-name brainstorm, compare how many distinct, usable ideas you get. For extracting a price, compare whether the answer exactly matches the supplied source. Begin with recommended defaults and improve the prompt or evidence when those are the problem. There is no universal best temperature for every model and task.
Brainstorming
Distinct ideas that fit the brief
Extraction
Exact match to source facts
Publishing
Useful writing with checked claims
Check your understanding
1. With probabilities 50%, 25%, 15%, and 10%, which tokens remain at top-p = 0.7?
2. Does higher temperature guarantee a more accurate answer?
3. Does top-p = 1 give every token an equal chance?
Reveal the answers
1. A and B: together they reach 75%, crossing 70%.
2. No. It changes the probability distribution; accuracy requires evaluation.
3. No. It disables nucleus truncation, while the remaining distribution can still be highly uneven.
Temperature reshapes probabilities.
Top-p restricts the candidate pool.
Neither guarantees truth.
Use documented defaults, test one change at a time, and judge the actual results.
Next lesson: Why AI hallucinates
We will explore why AI can produce confident but unsupported answers, how to recognize them, and how source-based checking helps.
Join the conversation: For Raju’s menu names, would you prioritize variety, consistency, or factual support? Explain your choice.
- Hugging Face: Generation configuration
- Hugging Face: Decoding methods and sampling
- Google Cloud: Content generation parameters
Examples and numerical exercises were created for this lesson. Consult the current documentation for your chosen model.
Comments
Post a Comment