Every number between your prompt and its answer

Here is a colour described by three numbers: how much red, green and blue. That list of three is a vector. Move the sliders: the numbers change the thing they describe.

Language models describe each piece of text the same way, just with far more numbers. Two separate tables are involved, and both are looked up with the same ID number:

• The vocabulary (in tokenizer.json) only maps IDs to text: ID 3575 is the text “␣capital”. It contains no numbers about meaning.
• The embedding table (in the model file) gives every ID its own list of 576 numbers. That list is what the model actually works with.

Why 576? It’s a size the model’s makers picked before training, called the hidden size. Bigger means more room to encode meaning, but every matrix grows with it, so the model gets slower and needs more memory. Small models use a few hundred; GPT-2 used 768, Llama 3 8B uses 4,096 and Llama 3 70B uses 8,192. Nobody decided what each of the 576 numbers means either. Training settled on values that make the model good at predicting text, so the numbers end up capturing things like “this is a city” or “this is plural”, spread across many positions at once.

A matrix is a grid of numbers, like a spreadsheet: so many rows, so many columns. Tensor is the catch-all word for a box of numbers with any number of directions. Files and code say “tensor” because they don’t care how many directions a box has.

The last row is the trickiest, so here is its shape drawn out. Picture one sheet of numbers per layer. On each sheet, every token is a column, and the column holds 3 heads × 64 = 192 numbers. Stack 30 sheets and you have the keys; the values are a second stack of the same shape. Chapter 10 shows the real numbers.

Almost all of the model’s work is this. Take a vector in. For each row of the matrix, multiply the row and the vector number by number and add up the results. That sum is a dot product, and each row gives one output number. The result is a new vector. Click an output to see its arithmetic, and edit the input numbers.

The sizes always follow one rule. A matrix with m rows and n columns needs an input of exactly n numbers, one per column, and gives back m numbers, one per row. Feed it several input vectors side by side and you get that many output vectors in one go. That is how the model handles a whole prompt at once.

To produce a single word, this model performs about 134 million of these multiply-and-add steps.

The numbers in the model’s matrices are its weights, also called parameters. Training adjusted them over trillions of words of text until the model got good at guessing the next word. When you use the model, which is called inference, the weights are frozen: nothing is learned and nothing changes. The 269 MB download is about 135 million of these numbers, and nothing else.

Running numbers through a model from input to output is a forward pass. That is all inference is, and all this page ever does.

Training adds a second direction. After each forward pass, the output is compared with the right answer: the word that actually came next in some real text. Then a backward pass (backpropagation) works out, for every single weight, which direction would have made the right answer a little more likely. Every weight gets nudged that way, and it repeats. SmolLM2 went through this with 2 trillion tokens of text. Try it on a model with 9 weights:

A real model does exactly this, just with 135 million weights instead of 9, and with the error passed back through all 30 layers instead of one. When training is finished, the weights are frozen and published. Using the model only needs the forward half: no comparing, no nudging. That’s why running a model is so much cheaper than training it.

A server running a model has two very different processors. The CPU handles text and decisions. The GPU does the giant multiplications. Each has its own memory, and moving numbers between places takes time.

The key fact for everything that follows: a modern GPU can multiply numbers much faster than it can fetch them from its own memory. Much of what makes inference fast or slow comes down to how many bytes have to move, not how much arithmetic there is. The map on the right (at the bottom on a phone) follows every chapter and lights up where each step happens.

A handful of sizes fixes every matrix in the model. For example, hidden_size of 576 means every token is a vector of 576 numbers, and num_hidden_layers of 30 means the tower in the 3D view is 30 layers high.

This file is a table of contents followed by the numbers themselves, packed back to back. The table of contents lists every matrix by name and shape. Laid out as a tree, it gives the model’s whole anatomy. The key thing to see: each of the 30 layers has its own copy of the same set of matrices. The shapes are identical, the numbers inside are different.

Adding it all up by type:

Pick any matrix below to see what it does and to look at its actual numbers.

Search the model’s vocabulary, or browse parts of it. On this page ␣ marks a space and ↵ a new line. Spaces are part of tokens: ␣cat (as in “the cat”) is a different entry from cat at the start of a line. The full list is in tokenizer.json on Hugging Face (a 2 MB text file), and Tiktokenizer lets you compare how different models split the same text.

First the text is cut at spaces and punctuation, keeping each space attached to the word after it. Chat markers like <|im_start|>, shown in black, are special tokens with fixed IDs: they tell the model who is speaking. Edit the prompt above and watch these update.

This is the part that isn’t obvious: the tokenizer does not search the dictionary for your word. It follows a recipe:

1. Break the word into single letters (strictly, bytes). Every single byte is in the dictionary, so this always works, even for emoji or made-up words.
2. Look at each pair of neighbours. The dictionary file also stores a ranked list of 48,900 glue rules, such as “Ġ + t → Ġt” as rule #0. They were learned in training by repeatedly gluing the most common pair in a huge pile of text. Apply the highest-ranked rule that matches.
3. Repeat until no rule matches. Every glued piece is itself a dictionary entry, so whatever is left has an ID.

Common words get glued all the way into one piece. Rare or made-up words run out of rules partway and stay as several pieces. This method is called Byte Pair Encoding. In this view a space appears as Ġ, which is how the dictionary file writes it. Try the examples:

Each final piece is a dictionary entry with an ID. This list of integers, 4 bytes each, is all that gets sent to the GPU.

How to read this: the strip on the left is the whole table, all 49,152 rows squeezed to fit, with teal marks at the rows your prompt uses. On the right, each line is one of your tokens, with its 576 numbers drawn as coloured bars: blue for negative, red for positive, paler for close to zero. The patterns look random to us. To the model they encode what the token means and how it is used.

One way to check that these numbers mean something: tokens used in similar ways get similar vectors. Pick a token to find the rows closest to it in the table.

Its trick is like clock hands. Picture each token’s vector as a clock hand. Before tokens are compared, every hand is turned by an amount that depends on the token’s position: the token at position 1 is turned a little, position 2 a little more, and so on.

When the model compares two tokens, it effectively looks at the angle between their hands. That angle depends only on how far apart the tokens are, not on where they sit in the text. Words three apart look the same to the model at the start of a sentence or a thousand words in. The widget below uses the model’s real vectors. Move the positions, then shift both tokens later and watch the angle between them stay put.

That repeated block is what makes this a transformer, the design behind GPT, Llama, Qwen, Claude and this model. Step through it with Next, or let it autoplay, to watch a short sentence go through it.

Why both halves? They do different jobs, and each needs the other:

• Attention moves information between tokens, but it only mixes and re-weights what is already there.
• Feed-forward transforms each token, applying patterns stored in its weights, but it can’t see the other tokens.
• Alternating them 30 times lets the model build up meaning step by step. Roughly, early layers sort out nearby words and grammar, middle layers combine facts, and late layers settle on the next word. The logit lens in chapter 7 shows this with real numbers: the right answer usually appears only in the last third.

The next two chapters zoom into each half. Chapter 9 replays this loop on your whole prompt at once, as a matrix with one row per token, with the real numbers.

Now zoom in on a single token and watch only the shape of its vector. It enters as a column of 576 numbers. Each step multiplies it by a weight matrix, and the shape rule from chapter 0 decides what comes out: (m × n) × (n × 1) = (m × 1). The table underneath keeps count of every matrix the vector passes through, and adds up to the whole model. After you generate, the pictures use your prompt’s real vector and the model’s real weights.

It works like a library search. Each token makes three new vectors by multiplying its own vector by three weight matrices:

• a query: what am I looking for?
• a key: what do I contain? This is like a book’s title.
• a value: what will I hand over if picked? This is like the book’s contents.

Each token compares its query with the key of every earlier token (a dot product). Better matches get a bigger share of its attention, with shares adding up to 100%. The token then takes that mix of the earlier tokens’ values. A token can never look at tokens after it, because when generating text those don’t exist yet.

Is this per layer? Yes. Each of the 30 layers has its own three matrices for this, plus a fourth (o_proj) that combines the results: 4 attention matrices × 30 layers = 120 separate matrices. They’re the four light tiles in every layer of the 3D model and the self_attn group in chapter 1’s anatomy. Every layer does the same steps with different numbers, so each layer looks for different things.

One more detail, because it explains the odd matrix sizes. The query isn’t used as one long list of 576 numbers. It is cut into 9 slices of 64, and each slice does its own attention, separately from the others. Each slice is called a head. Nine heads means nine independent “searches” per layer: one head might track the previous word, another the subject of the sentence. That’s why the grid below changes when you change Head.

Keys and values are cut the same way, but there are only 3 slices of them (3 × 64 = 192 numbers). Each key/value slice is shared by 3 query heads. That saves memory, and it’s exactly why k_proj and v_proj are 192 × 576 while q_proj is 576 × 576. Here are the real vectors for the last token of your prompt in layer 0, each number a coloured cell:

How to read this: each thin bar is one of the 1,536 intermediate numbers for the last token of your prompt, in the layer you pick. Most sit near zero and a few spike. Each spike is a learned pattern that recognised something in this context and fired.

Neither attention nor the feed-forward step overwrites a token’s vector. Each adds its result to the vector instead, like adding notes to a running total. That running vector is the residual stream, and it carries everything the token has picked up so far. How to read this: each line is one token of your prompt, showing how long its vector gets after each layer. The highlighted line is the last token, the one that will predict the next word.

The running total can be read at any point to see what the model would say if it stopped there. This trick is called the logit lens. Early layers produce nonsense, and the real answer usually appears in the last third.

You control the dice:

• Temperature below 1 makes the favourite even more likely (safer, more repetitive). Above 1 evens the odds (more varied, more mistakes). At 0 the model always takes the top choice, called greedy.
• Top-k only allows the k highest-scoring tokens.
• Top-p only allows the smallest group of top tokens whose chances add up to p.

Click any word of the answer above to see the choice behind it. Click another candidate below to swap it in and continue from there.

Only the winning ID, a single number, goes back to the CPU. The CPU reverses the tokenizer: it looks up the dictionary entry, turns it into bytes, and the bytes into characters.

How to read the pictures. Each picture is a matrix: one row per token of your prompt, labelled on the left, and one column per number. Red is positive, blue is negative, white is near zero. The bottom row, highlighted, is the last token, the one whose final row picks the next word. Hover over any cell to read its token, column and value. Shapes follow the rule from chapter 0, with T in amber. Only one step, attention, lets the rows talk to each other; everything else works on each row separately.

On every pass, the new token needs to look back at all earlier tokens (chapter 6), using their keys and values. Those never change once computed, so instead of recomputing them on every pass the model keeps them, like notes, in GPU memory. These notes are the KV cache. Each pass adds one column of notes for the new token, in every layer.

How to read this: each column is one token and each row is one layer. Every small cell holds that token’s key and value for that layer. Amber cells came from your prompt; pink cells were added one by one as the answer was written.

Here are the actual numbers inside one layer’s cache: the 4-D shape from chapter 0, with a single layer picked out. Each column is one token. Each column holds that token’s key (3 heads × 64 numbers) and, on the right, its value. Watch a new column appear for every word the model writes.

Without the cache, every new word means redoing the work for the whole text so far. Untick the box and press Generate (or Continue). Each bar is the real time one step took in this tab.

Reading the prompt is different. The chef fetches each ingredient once and uses it for all the prompt’s tokens at once, so there is lots of chopping per trip. That is compute-bound, and it’s why prefill handles hundreds of tokens in the time decode writes a few.

Servers fix the walking problem by cooking many orders per trip: they batch many users’ next words into one pass, so each weight fetched is used for all of them. Drag the sliders and watch the dots move toward the flat “compute” part of the chart, where the chef finally stops waiting.

This page’s own multiply code reuses each fetched weight for up to 4 tokens at once. Time it doing one token at a time (like decode) and 32 at a time (like prefill):

Notes are kept in blocks of 16 tokens. A block is reused only if it and everything before it match exactly. Change one word early on and every block after it has to be recomputed. That’s why production apps keep their system prompt identical, byte for byte.

Try it: run a question, then press “Ask a different question”. Only the question changes, so the shared blocks are reused and the time to the first word drops. The default system prompt is short, so only one block can be shared. Switch to the longer one to see a bigger effect.