How are ChatGPT and Claude actually trained? Pretraining, human feedback, and why they are so polite

Pretraining is the stage people have heard of. A model is shown a vast amount of text, most of the public internet plus books and code, and trained on the one move from lesson one: predict the next token. Months on thousands of GPUs. The result is a base model, and a base model is strange. Ask it a question and it may continue with ten more questions, because on the internet questions are often followed by questions. Ask for help and it may write a forum post complaining. It has read everything and wants nothing.

The base model is where almost all the knowledge lives. The next two stages add very little knowledge. They change behaviour.

Supervised fine-tuning is short. People write a few tens of thousands of example conversations, the kind of answer a helpful assistant should give, and the model is trained on those. Now it answers questions instead of continuing them. It has a format.

That gets you most of the way to a chatbot and none of the way to a good one. Written examples cannot cover every case, and the model has no sense of better or worse, only of "like the examples". The third stage adds that sense, and it is the one worth understanding.

RLHF stands for reinforcement learning from human feedback. The mechanism is simpler than the name. The model writes several answers to the same prompt. A person ranks them. A second model, the reward model, is trained to predict those rankings. Then the chat model is trained to produce answers the reward model scores highly. Repeat with thousands of prompts and raters.

In the 2022 paper that put this into ChatGPT, OpenAI's raters preferred a 1.3-billion-parameter model trained this way over the raw 175-billion-parameter GPT-3. A hundred times smaller, and people liked it more. That is how much behaviour matters compared with knowledge.

The lab makes you the rater. The "model" is a simulation with a handful of style knobs, and the reward model is a few numbers that learn from your rankings. The real system is the same loop with a hundred billion parameters. Rank honestly, then rank the way a tired person would at the end of a shift, and watch what happens to the knobs.

If you rewarded the kind, agreeable answers, you saw "Flatters you" climb and "Says the hard thing" fall. That is not a quirk of our toy. It is what happened at scale. People, on average, rank the answer that agrees with them above the answer that corrects them. The reward model learns that. The chat model learns to please the reward model. The result is a model that opens with "Great question!" and tells you your cat-mood app is a wonderful idea.

In April 2025 OpenAI shipped a ChatGPT update that had leaned harder on thumbs-up feedback and rolled it back four days later because the model had become, in their own word, sycophantic. Their post-mortem says the thumbs-up signal "can sometimes favour more agreeable responses". One Reddit user counted 1,100 times a model told them "great question" in a year. That is RLHF working exactly as designed, on the wrong target.

Anthropic's answer, called constitutional AI, replaces part of the human ranking with a written set of principles the model uses to critique and revise its own answers. It reduces how much the model learns from individual raters' moods. It does not remove the problem; it moves it to whoever writes the principles.

The same stage decides what the model refuses. Raters mark some answers as harmful, the reward model learns a boundary, and the chat model learns to stay inside it. The boundary is learned from examples, not written as rules, which is why two people with nearly identical prompts can get a refusal and an answer. Nothing about your account is involved. You are at the fuzzy edge of a learned shape.

Labs tune this constantly. In the lab, if you ranked the refusal first a few times, "Refuses" climbed and the model started declining everything. Rank it last and it stops. The real dial is the same, just with thousands of hands on it.

One misconception needs killing cleanly. When you chat with ChatGPT or Claude, the model's weights do not change. Training is a separate, expensive process that happens offline, on data chosen by the lab. Your conversation may be stored and may be used in a future training run, which is what the privacy switch in our how-to controls, but the model you are talking to today is frozen. It is not learning from you as you type. If it seems to, that is memory and context, which the context lesson covers.