An honest toy · next-token sampling
Every word a model writes is a roll of a weighted die.
A language model does not "know" the next word. It scores every word it could say next, turns those scores into probabilities, and rolls a weighted die. Pick a sentence, reshape the odds, then roll it yourself and watch the line finish.
Illustrative model, not a live model call. The distributions are hand-authored, the same way How Money Moves is a toy of the real banking rails.
Pick a sentence, drag the temperature, then roll the die.
The sentence so far
What could come next
temperature 1.00The weighted die · faces sized by probabilityroll to sample
The marble lands at a random point from 0 to 1. Whichever face it falls on is the chosen word, so a wider face gets picked more often. Cut faces and the rare-word tail are not on the die.
▮ Greedy (always pick the top face) would choose its top face here.
What you are seeing
Four knobs sit between a model's scores and the word it says.
For every possible next word, the model produces a raw score. Softmax turns those scores into probabilities that sum to one, the heights you see in the bars. Temperature divides the scores before that step: turn it down and the gaps between scores stretch, so the top word runs away with all the probability; turn it up and the gaps compress, so even unlikely words get a real shot. At very low temperature the die has one giant face, which is why a factual prompt barely changes no matter how often you roll.
Top-k keeps only the k highest faces and removes the rest before rolling. Top-p, also called nucleus sampling, keeps the smallest group of faces whose probabilities add up to p, so it trims more aggressively when the model is confident and barely at all when the model is unsure. Both exist to cut off the long, weird tail without forcing the model to always pick its single favorite. With sampling off (greedy), none of this matters: the model just takes the top face every time, which is safe, repeatable, and a little lifeless.
This is a teaching toy. A real model scores tens of thousands of tokens at once, not the handful shown here, and its numbers come from billions of parameters rather than my hand. The grey "rest of the vocabulary" bar stands in for that whole long tail. The mechanism, though, is exactly this: score, soften, trim, roll. That's the whole trick behind every chatbot you have ever used.
Change one thing, the temperature, and the same prompt swings from a careful clerk to a reckless poet. The intelligence is in the scores; the personality is in how you roll.