How does ChatGPT work? It guesses the next word, one token at a time

Here is the whole of ChatGPT in one sentence. Given the text so far, score every possible next piece of text, pick one, add it to the text, and do it again.

That is it. There is no plan for the paragraph. There is no lookup of facts in a database. There is no understanding in the way you understand this sentence, and also no trickery: just a very good guess about what comes next, made again and again, fast enough that it looks like typing.

People hear this and say "so it's just autocomplete". Hold that thought until the end of the lesson. You will operate the move yourself, on a real model, and then decide whether "just" is the right word.

Before any guessing, the text is chopped into tokens: pieces that are sometimes a word, sometimes part of a word, sometimes a single character or a piece of punctuation. The chopping rules were learned from a mountain of mostly English text, so English gets efficient pieces. Everything else gets chopped finer.

OpenAI's own rule of thumb is that one token is about four characters of English, so 100 tokens is roughly 75 words. That rule breaks the moment you leave English. Try the Hindi example in the lab and compare it with the English one above it. Same sentence, same meaning, more than five times the tokens. You pay per token, the model's memory is counted in tokens, and its "attention" is spread across tokens. So a Hindi speaker gets less of everything for the same price. This is the single most important thing about AI that nobody in India is told.

Now the guess. The model takes all the tokens so far and produces one number for every token in its vocabulary. GPT-2, the model in our labs, has 50,257 of them. Those numbers are turned into probabilities that add up to 100 percent. The bar chart you are about to see is the real output of a real model, not a drawing.

Three things to notice. First, the model is rarely sure: after "The capital of France is", the top guess is "the", not "Paris", because in the text it learned from, that phrase is followed by all kinds of things. Second, the probabilities are spread over thousands of tokens; the eight bars we show hold most of the bets but never all of them. Third, the model is small and old. GPT-2 was released in 2019 and has 124 million parameters; today's models have a thousand times more and are much better at this. The move is identical.

You just used the most misunderstood setting in AI. Temperature does not make the model smarter or dumber. It reshapes the odds before the dice are rolled. At low temperature the top choice gets nearly all the probability, so you get the same safe answer every time. At high temperature the odds flatten and the long tail of unlikely tokens gets real weight.

Watch the percentages in the lab as you slide it. At 0.1 the top word is close to certain. At 2.0 the eight bars together hold much less, and "roll the dice" lands on something strange. That is not a bug. It is the same model with a different dial. And it is the honest explanation of one kind of hallucination: a low-probability token got picked, and then every token after it had to make sense of it.

Chat apps hide this dial and set it somewhere in the middle. Developers can set it. When someone says a model "got creative", often they mean the dial was up.

People are surprised that the same question gives different answers. Now you know why: the pick is a dice roll weighted by the odds. Press "roll the dice" a few times from the same start and you get different sentences, each one plausible. Only at the very lowest temperature does it become deterministic, and even then small differences in the text so far change the odds.

This is also why "regenerate" works. You are not asking the model to think harder. You are rolling again.

Yes, mechanically. Every token comes from the move you just operated. And no, in the way people mean it. On the r/singularity thread that argues this out every few months, the top answer puts it well: next-word prediction describes the mechanism, not what the mechanism can do. To predict the next token of a chess game well, you have to model chess. To predict the next token of a legal argument, you have to model the argument. Scale the move up a thousand times and train it on most of the written internet, and the guesses start to contain a working model of the world the text describes.

What the move cannot do is also now clear. It cannot check a fact against a source, because there is no source, only odds. It cannot know that it does not know, because a low-confidence guess still produces a confident-looking sentence. Both of those get their own lesson.