Top P and Temperature

Temperature

The usual order in most implementations (OpenAI, Hugging Face, etc.) is:

1. Apply temperature first -> this reshapes (sharpens or flattens) the probability distribution.

• If temperature < 1, high-probability tokens get boosted (distribution sharper).
• If temperature > 1, probabilities spread out more evenly (distribution flatter - helps less likely tokens).

1. Apply top-p (nucleus sampling) next -> from that temperature-adjusted distribution, collect the smallest set of tokens whose cumulative probability >= p, then sample from that set.

Top P

The model produces a probability distribution over all possible next tokens. Example (ASSUME top p is set to .8):

• Token A -> 0.40
• Token B -> 0.30
• Token C -> 0.15
• Token D -> 0.10
• Token E -> 0.05
• (These numbers sum to 1.0.)

With top_p = 0.8, we build the smallest set of tokens whose cumulative probability >= 0.8:

A (0.40) + B (0.30) = 0.70 (still below 0.8).

Add C (0.15) -> total = 0.85 >= 0.8.

So the nucleus set is {A, B, C}, which made up .85

This means:

• The most likely token (A = 0.40) is always included.
• What gets excluded are the least likely tokens (D, E in this case).

Renormalization

We rescale {A, B, C} to sum to 1:

• A -> 0.40 / 0.85 = 0.47
• B -> 0.30 / 0.85 = 0.35
• C -> 0.15 / 0.85 = 0.18

Then sample randomly

Now the model draws one token at random from this set, weighted by these renormalized probabilities.

• A has a 47% chance
• B has a 35% chance
• C has a 18% chance

To do this the model generates a random number between 0 and 1

• if the number is 0.0 - 0.47 then token A is selected
• if the number is 0.48 - 0.82 then token B is selected
• if the number is 0.83 - 1 then token C is selected