Why does AI hallucinate? Why ChatGPT is confidently wrong and can't say 'I don't know'

When ChatGPT told people with banned Instagram accounts to email a magazine for help, giving the magazine's real address, it was not deceiving anyone. Lying needs a truth you are hiding. The model had none. It had learned that text about account problems is often followed by text about who to contact, and it produced the likeliest contact, and the likeliest contact was wrong.

Go back to the move from lesson one. Score every next token, pick one, repeat. Nowhere in that loop is a step that checks a claim against a source. There is no source. There are only the odds the model learned. So the honest name for a hallucination is a plausible guess that happened to be false, delivered in the same voice as a plausible guess that happened to be true.

This is the part people find hardest to accept. Surely the model could just say it is unsure? It could, and it has been trained not to.

In September 2025 OpenAI published a paper with a plain title, "Why Language Models Hallucinate". Its argument is about grading. Nearly every test used to measure and train models scores a right answer as one point and both a wrong answer and "I don't know" as zero. Under that scoring, guessing always beats abstaining, the way it does in an exam with no negative marking. A model that is trained and selected on those tests learns to guess. The paper's example is a model asked for one of the authors' birthdays: it gave three different confident dates, all wrong, rather than say it did not know.

The paper's fix is the one every Indian student knows from competitive exams: negative marking. Score a wrong answer below an abstention and the incentive flips. Some labs have started reporting abstention rates alongside accuracy. The grading is changing slowly; the habit it trained is already in the models you use.

There is a precise word for "knowing how sure you should be": calibration. A calibrated forecaster who says 70 percent is right about 70 percent of the time. You can plot it: confidence along the bottom, how often you were actually right up the side, and a perfect forecaster sits on the diagonal.

Before you look at a model's curve, get your own. Twelve questions, each with how sure you are. Be honest about the confidence; the whole point is in the gap.

The grey lines in your chart are redrawn from figure 8 of the GPT-4 technical report, and they are the most important chart on hallucination that almost nobody has seen. The pretrained model, before any human feedback, was almost perfectly calibrated on a large exam-style test. When it said 80 percent, it was right about 80 percent of the time. Then the model went through the human-feedback stage from the last lesson, and the curve bowed away from the diagonal. The report's caption says it plainly: the post-training "hurts calibration significantly".

Think about why, with the RLHF lesson in mind. Raters prefer confident answers. The reward model learns that. The chat model learns to sound sure. The knowledge of its own uncertainty was there in the base model, and the training to be likeable paved over it. Hallucination is not only a knowledge problem. It is partly a manners problem we trained in.

Three things work, and you have seen two of them in this Academy. Search grounding works because it gives the model a source to quote instead of odds to sample: the loop now has text in its window that says the fact. Lower temperature works a little, because it stops the dice landing on a tail token that the rest of the sentence then has to justify. And asking for the exact quote with a link works because it turns a vague recall task into a copying task, which models are good at, and gives you something to check.

What does not work is asking the model whether it is sure. That question goes through the same loop and gets the same confident voice. The r/OpenAI thread titled "Why does ChatGPT never know when to say 'I don't know'" has the right top answer: it does not have access to its own state in a way it can report. The calibration is in the probabilities, which the chat window hides from you, not in the words.

So the practical rule from our how-to stands, and now you know why it is the rule: for anything with a number, a name, a date or a link in it, make the model show its source and open the source yourself.