How does a transformer know which words matter? Attention, shown for real

In the last lesson the model scored the next token using "the text so far". That hides a hard question. The text so far might be 500 words. Which of them matter for the next guess?

Take "The animal did not cross the street because it was too tired." To guess what comes after "it was too tired", the model needs to know "it" means the animal. Swap the last word for "wide" and "it" means the street. Nothing else in the sentence changed. An older kind of model read text one word at a time and tried to carry everything forward in one memory, which blurred by the time it reached "it". The 2017 paper from Google, "Attention Is All You Need", threw that away. Every word gets to look directly at every earlier word and decide, for itself, how much each one matters. That lookup is attention, and it is why the paper's architecture, the transformer, is behind every model you have used since.

A transformer has many of these lookups running side by side. Each is called a head. GPT-2, the model in our lab, has 12 layers with 12 heads each, 144 in total. Each head produces a table: for every word, a share of attention over the words before it, adding up to 100 percent. The numbers in the lab are those tables, read straight out of the model.

Two rules to know before you hover. A word can only attend to itself and the words before it, because this model is trained to predict what comes next and is not allowed to peek. And every head learned its own habit from the data; nobody assigned them jobs. Some of the habits are easy to name.

The first preset is a head that almost always looks at the word just before. Hover anything and the arc goes one step back. This is how a model keeps word order without being told what order is. "Not cross" and "cross not" look different to it because of heads like this.

The second is the one that earns the lesson. In "Rahul gave his sister the book she had asked for", hover "she". The head sends half of its attention to "sister" and under a third to "Rahul", with the rest scattered. That is pronoun resolution, the thing we said older models blurred, happening in one head of a small 2019 model, learned from nothing but predicting next words.

The third looks like a failure and is not. Many heads dump most of their attention on the first word of the sentence whatever you hover. Researchers call it an attention sink. The head has nothing useful to look at for this word, and the first position is where it parks. Once you know this you will see it in every attention picture ever published.

Now drag the layer and head sliders yourself. Most heads are messy and hard to name. That is the honest picture: a few heads do readable jobs, the rest do their part in ways no one has fully decoded. Interpretability researchers at Anthropic and elsewhere spend their careers on exactly this.

The biggest wrong mental model is that the model reads left to right, one word at a time, like a person. It does not. All the words in the window are processed at once, and attention is the mechanism that lets any word reach any earlier word in one step, however far back. That is why the 2017 paper could train on eight GPUs in three and a half days: the work parallelises. It is also why context windows are measured in tokens and why they cost so much to grow: every token attending to every earlier token is a cost that grows with the square of the length.

The lesson on context windows picks that up. For now, hold on to the picture: not a reader moving along a line, but a sentence where every word is looking at every other word, in 144 different ways, at the same time.

"Attention Is All You Need" was published in June 2017 by eight researchers at Google. It proposed the transformer for translation. It has been cited more than 250,000 times and is in the top ten most-cited papers of this century. GPT stands for Generative Pre-trained Transformer. Claude and Gemini are transformers. So are the image models, mostly, and the models that write code and the ones that fold proteins.

The paper did not predict any of that. It reported that a model built only from attention translated English to German a little better and much faster than the models of the day. Everything else came from the fact that the same mechanism, given more data and more compute, kept getting better, and nobody has found the ceiling yet.