A language model does not produce text — it produces a probability distribution over the next token, thousands of candidates each with a score. Sampling is how one candidate gets picked, and the sampling parameters decide how adventurous that pick is.
Temperature: the adventure dial
Low temperature sharpens the distribution (the top candidate almost always wins); high temperature flattens it (unlikely tokens get a real chance). Same prompt, three temperatures:
Describe rain in one sentence.
Rain is water falling from clouds in drops.
Rain drums softly on rooftops, turning streets into mirrors.
Rain: the sky rehearsing its silver piano across a city of umbrellas.
Rules of thumb:
- 0 – 0.3 for extraction, classification, code — anything with a right answer.
- 0.6 – 0.9 for conversation and general writing.
- 1.0+ for brainstorming and deliberately weird ideas.
Top-p is the other common dial: instead of reshaping probabilities, it cuts the candidate list to the smallest set covering p of the probability mass. In practice you tune one of the two, not both.
Why randomness is still controlled
Sampling can sound like the model is rolling dice, but the dice are loaded by the probability distribution. At temperature 0, the model keeps choosing the strongest next token, so repeated runs are stable enough for extraction and evaluation. At higher temperatures, the model is still not free-associating from nowhere: it is allowed to pick from weaker candidates more often. That is why a high-temperature answer can be lively and still stay on topic, until the temperature gets so high that weak candidates start steering the sentence away from the task.
Use this distinction when you read receipts and debug output. If a low temperature answer is wrong, the problem is probably in the prompt, the context, or the model’s knowledge — rerunning the same request is unlikely to magically fix it. If a high-temperature answer is merely odd, the prompt may be fine; you asked the sampler to explore. The fix is to lower temperature, ask for a stricter format, or run several candidates and choose one with a checker.
Top-p gives you a different kind of safety rail. A low top-p value says: ignore the long tail of weird candidates and sample only from the most plausible cluster. That can be useful for chat where you still want natural phrasing but do not want the model reaching too far. For most beginner work, start with temperature. Bring in top-p only when you can explain what behavior you are trying to change.
Try it yourself
Move the sliders and feel the difference. Ask the same question twice at temperature 0, then twice at 1.5:
Check your understanding
You are building a JSON extractor. Which temperature fits best?