A sentence goes in
You type six words. An answer comes back — structured, plausible, sometimes correct. Nothing magical happened in between. Something fairly strange did.
Take one question and follow it all the way down.
Why is the sky blue?
To you this is a sentence. To the model it is not one. It is a sequence of numbers first, and nothing else will ever reach it.
The sentence comes apart
Before any computation, text is broken into tokens: pieces that are neither quite letters nor quite words. Frequent words are often a single token; rare ones shatter into several.
Tokenisation
- Why
- is
- the
- sky
- blue
- ?
6 tokens
Each chip is a token, numbered by position. Try your own sentence — the token count rarely moves the way you expect.
Conceptual representation. The split shown here follows a simple teaching rule — words, punctuation, and a few common suffixes. Real tokenizers learn their vocabulary from a corpus, and two models will cut the same sentence differently.
Text version
The sentence “Why is the sky blue?” splits into six tokens:
Why, is, the, sky,
blue, ?. The rule used here separates words,
isolates punctuation, and detaches a few frequent suffixes from long words.
Something is already lost. The model will never see “sky”. It sees token number three, which happens to carry that name for our convenience.
Every token becomes a position
A token on its own means nothing. Each one is mapped to a vector — a list of numbers — and it is that list, not the word, that travels through the model.
The vector itself is not the interesting part. What matters is that vectors for related words end up near one another. That proximity is not declared: it emerges from training, because those words appear in comparable contexts.
Representation space
Choose a word
king Nearest
- 1 queen
- 2 man
- 3 woman
Select a word by click, tap or keyboard: its three nearest neighbours appear.
Conceptual representation. Real representations have hundreds, often thousands of dimensions, and no axis carries a name. This plane stages the only thing that matters here — that “near” means something.
Text version
Ten words in three groups: king, queen,
man, woman together; apple,
banana, orange apart from them;
car, truck, bicycle lower down. The
immediate neighbours of king are queen,
man and woman — never a fruit.
Position changes everything
Two sentences, exactly the same tokens:
dog bites man
man bites dog
A bag of identical vectors cannot tell these two worlds apart. So the information of where a token sits in the sequence has to be injected into its vector.
It is an addition, literally: a representation of the position is added to the representation of the token. From there, “dog in first position” and “dog in third position” are no longer the same object.
Every word looks at the others
Here is the central mechanism. Take a sentence with an ambiguous pronoun:
Attention Explorer
Select a word
it
Select an outlined word. The arcs and bars show where that word draws its information from.
Conceptual representation. The weights shown are illustrative: they are set by hand to make visible the relation the sentence actually carries. They are not measured weights from any named model.
Text version
For the word it, the strongest weights point to
animal (0.62), then street (0.13),
cross (0.07) and tired (0.06). For
tired, they point to animal (0.35) and
it (0.30).
“It” refers to the animal, not the street. No grammar rule was written to establish that: the model learned to carry a large share of “animal” into the representation of “it”. That is attention — a learned, weighted average of what the other positions have to offer.
Three roles: asking, offering, carrying
To compute those weights, every token plays three roles at once. The easiest way in is to read them as three questions.
Query — what am I looking for? For “it”, something like: a noun that could plausibly be tired.
Key — what do I advertise? Each token exposes a label saying what it can satisfy.
Value — what do I pass on if I am selected? The information that actually flows forward.
The attention weight is the match between a query and a key. What travels is the value.
Under the hood
Each token produces its three roles by linear projection of its
representation X:
Q = X·Wq K = X·Wk V = X·WvThe score between a query and every key is a dot product, scaled and then turned into a distribution:
Attention(Q, K, V) = softmax( Q·Kᵀ / √dk ) · VDividing by √dk — the square root of the key dimension — keeps the dot
products from growing large enough that the softmax saturates and lets only a
single token through.
The equation arrives here, not at the start: it formalises the intuition, it does not replace it.
Several readings in parallel
One set of weights is not enough. A sentence carries several relations at once — who does what, what refers to what, what is nearby, what holds the structure. So the model computes several attentions in parallel, over different projections of the same sentence.
Multi-head attention
Selected word: “it”
it
This projection puts nearly all of its weight on the subject of the sentence.
Same sentence, same selected word, four different projections. Switch between heads: what each one keeps changes.
Conceptual representation. The four patterns are built for the explanation. Heads are learned projections: they do not have permanently assigned, human-readable jobs, and the labels below are a reading convenience, not a property of the model.
Text version
Four projections of the same word it: the first points
overwhelmingly at animal (0.73); the second stays on
was and tired; the third targets
street (0.45) and cross (0.29); the fourth
spreads roughly evenly across the sentence.
The heads’ outputs are concatenated, then recombined. The sentence comes out carrying several readings at once.
Attention is only one part
Step back one level. Attention lives inside a block, and a block holds more than that: a normalisation, a dense network applied to each position independently, and residual connections that let information bypass every stage.
One block, then the stack
One block
- INPUT
- ATTENTION
- ADD / NORMALIZE
- MLP
- ADD / NORMALIZE
- OUTPUT
The stack
- 01
- 02
- 03
- 04
- N
The number of blocks varies by model, from a few dozen to a great many more. Each block receives the previous one's output: a token's representation is reworked once per layer.
On the left, the inside of one block. On the right, the same block repeated — that is the model's depth.
Text version
One block runs: input, attention, add and normalize, dense network, add and normalize, output. That block is repeated N times, each one’s output becoming the next one’s input.
This is where most of the computation lives. Attention decides what talks to what; the rest of the block decides what to do about it.
Only one thing comes out: the next token
At the top of the stack the model produces neither a sentence nor an idea. It produces a probability distribution over its whole vocabulary, for one token — the next one.
Generation
Context
The capital of France is
Candidates for the next token
- Paris 0.78
- located 0.05
- the 0.03
- a 0.02
- also 0.02
0 / 3
Press to generate a token. It joins the context, and everything starts again from the lengthened text.
Conceptual representation. No model runs on this page: the continuation shown and its probabilities are written in advance. What is faithful is the shape of the process — a distribution, one token kept, a context longer by one.
Text version
From “The capital of France is”, the most probable candidate is
Paris (0.78), far ahead of located (0.05). Once
Paris is kept, it enters the context and the computation
restarts: the next candidates are then a comma (0.31) or a full stop
(0.29).
Everything else follows from this. Long text is not produced in one go: it is this loop, repeated, where each output becomes an input. That is autoregressive generation.
So where does reasoning come from?
If the model only ever predicts the next token, how does it solve a problem that takes several steps?
Compare two trajectories. In the first, the question is asked and the answer drops: all the work has to fit in a single pass. In the second, the model first generates intermediate text — decomposing, testing, correcting — and that text, by entering the context, becomes data the rest of the computation can lean on.
The second trajectory adds no new organ. It adds generated computation before committing, and that alone changes what the model can carry through.
On reasoning tags
Some systems explicitly separate that intermediate production from the final answer, using tags of the form:
<thinking>
…
</thinking>These are conventions — of interface, of training, or of runtime, depending on the system. They are not a Transformer primitive, and in the architecture described here there is no separate compartment in which the model “thinks” before speaking. The mechanism is unchanged: token generation over a growing context.
What can go wrong
What this machine does not do
It has no memory between conversations. What is not in the context does not exist, and the context is finite.
It consults no source while answering. What it produces comes from its parameters and from what you handed it — hence statements that are perfectly well formed and perfectly wrong.
It has no internal measure of its own certainty that you can read. A high probability on the next token is not a guarantee about the substance; it is a statistic about the form.
And above all: it does nothing. No file opens, no message leaves, no machine stops. It writes.
A LLM generates. It does not act.
This is the most useful dividing line in the whole journey, and the one most often crossed without saying so.
A language model turns a context into next tokens. It decides nothing, triggers nothing, verifies nothing. Everything that looks like action comes from what was built around it.
So what happens if you give it memory, tools, and a loop?
The model
- Context
- Generation
Tools
Loop
An agent
- Context
- Generation
- Tools
- Loop
It produces text, and nothing else moves. Give it something to call and something to repeat with, and that text becomes an action.
You are here
What happens once you give this model memory, tools and a loop?