Parameters, Tokens and Embeddings

47.0 What this chapter gives you#

1. You will be able to say exactly what one parameter is: a single number in a single position of a single table inside the model.
2. You will be able to take an architecture description and compute the parameter count by hand, and say which parts of the model hold the billions.
3. You will be able to tell parameters apart from hyperparameters, activations, context and tokens, and use all five words correctly in one sentence.
4. You will be able to turn a parameter count into a memory figure, for inference and for training, and explain the gap between them.
5. You will be able to explain quantization, name the common formats, and work out whether a given model fits on a given machine.
6. You will be able to say honestly why parameter count is a weak predictor of how useful a model is, and cite the evidence for that.
7. You will be able to explain byte-pair encoding by working it through by hand on a tiny corpus, and name the three other tokenizer families.
8. You will be able to predict roughly how many tokens a piece of text costs, and explain why Hindi costs several times more than English for one meaning.
9. You will be able to explain why a model cannot count the letters in a word, and why it fails at arithmetic on long numbers, from tokenization alone.
10. You will be able to explain what an embedding is, what similarity between embeddings means, and what a context window really limits.

47.1 What a parameter actually IS in a language model#

PLAIN47.1.1 in simple words#

1. A language model is, on the inside, a very large pile of numbers.
2. Nothing else. No sentences stored. No facts written down as text. Numbers.
3. One of those numbers is called a parameter.
4. Chapter 46 fitted a straight line to five points. That line had exactly two parameters: a slope and an intercept, both found by gradient descent, which is the process of nudging numbers downhill until the error stops falling.
5. A language model is the same idea with the count raised enormously. Two parameters becomes eight billion, and the shape of the thing being fitted becomes a network of layers instead of a line.
6. Everything else in this section is that one sentence unpacked.
7. So a parameter is not a setting you choose. It is not an option in a menu.
8. It is one plain number, like -0.0113 or 0.0392, sitting in one fixed slot.
9. The slots are arranged in rectangles, called matrices. Rows and columns.
10. A parameter has an address: this matrix, this row, this column.
11. When the model was trained, every one of those numbers was slowly adjusted until the model got good at predicting the next piece of text.
12. When training finished, the numbers were frozen and written to a file.
13. Every time you use the model, it reads the same frozen numbers.
14. Your question does not change them. Your conversation does not change them.
15. When somebody says a model has 7 billion parameters, they mean the file contains 7 billion such numbers.
16. That is the whole idea. A parameter is one learned number in one slot.

PLAIN47.1.2 a picture in your head#

1. Picture an enormous sound mixing desk, the kind in a recording studio.
2. A mixing desk has knobs. Each knob controls how loudly one thing is heard.
3. Turn one knob and the guitar gets louder. Turn another and the drums fade.
4. Now imagine a desk with seven billion knobs instead of forty.
5. Each knob is set to some exact position, like 0.0392 out of a range.
6. Training was a very long process of nudging every knob a tiny amount, over and over, until the whole desk produced a good sound.
7. Shipping the model means writing down every knob position in a file.
8. Loading the model means setting all seven billion knobs back to those exact positions.
9. Running the model means pushing sound through the desk without touching a single knob.

Where this comparison breaks: on a real mixing desk, a human knows what each knob does, and there is a label under it. In a model, almost no single parameter has a meaning you can name. Its effect only exists in combination with millions of others. Also, a mixing desk has knobs a person turns during a performance. Model parameters are never turned during use. They are turned only during training, and then never again unless the model is retrained.

PLAIN47.1.3 a worked example#

1. Here is a genuine kind of object from inside a model: a weight matrix.
2. This is a four-by-four corner of one, with values rounded for printing.

A 4 x 4 corner of one weight matrix, values rounded

          col 0     col 1     col 2     col 3
row 0   -0.0113    0.0271   -0.0004    0.0158
row 1    0.0392   -0.0087    0.0221   -0.0310
row 2   -0.0056    0.0143    0.0079    0.0002
row 3    0.0018   -0.0225   -0.0166    0.0407

3. Look at row 1, column 2. The value there is 0.0221.
4. That is a parameter. That one number. Not the row. Not the matrix. That.
5. Its full address inside a real model would be something like: layer 7, the query projection matrix, row 1, column 2.
6. There are 16 parameters in this little corner, because it is 4 rows times 4 columns.
7. The real matrix this corner came from would be 4096 by 4096.
8. That is 4096 x 4096 = 16,777,216 parameters in that one matrix alone.
9. A model of this size has hundreds of matrices like it.
10. Every one of those numbers started as a random value before training, and ended up at its final value because training pushed it there.
11. Nobody chose 0.0221. Nobody can explain 0.0221 on its own.

PLAIN47.1.4 what is really happening inside#

1. The model’s job at each step is to turn a list of numbers into another list of numbers.
2. The main operation it uses is multiply-and-add, done many times at once.
3. Take an incoming list of 4096 numbers. Call it the input vector.
4. Take one column of the weight matrix, also 4096 numbers.
5. Multiply them pairwise and add the results up. That gives one output number.
6. Repeat for every column. That gives the whole output vector.
7. So each parameter is a multiplier. It says how much one incoming number contributes to one outgoing number.
8. The parameter at row 1, column 2 says: take input number 1, multiply it by 0.0221, and add that into output number 2.
9. That is literally all a weight does. It is a multiplier in one cell of a giant multiply-and-add.
10. There is a second, smaller kind of parameter called a bias.
11. A bias is added rather than multiplied. It shifts an output up or down by a fixed amount regardless of the input.
12. Biases are also learned during training, and are also parameters.
13. Some model families use biases everywhere, some use almost none.
14. When the model runs, it walks through layer after layer doing this. The parameters never move. Only the numbers flowing through them change.

TECHNICAL47.1.5 the engineer’s version#

1. A parameter is one scalar element of a learnable tensor, updated by the optimizer during training and fixed at inference time.
2. In PyTorch terms, a parameter is one element of a tensor wrapped in torch.nn.Parameter, which is a tensor with requires_grad=True that is registered in the module’s parameter list.
3. The words weight and parameter are used interchangeably in practice, and in almost every sentence you will read they mean the same thing.
4. The precise distinction is that a weight is a multiplicative parameter in a linear map, while a bias is an additive parameter. Both are parameters. So all weights are parameters, but not all parameters are weights.
5. Normalization scale and shift terms, often called gamma and beta, are also parameters. So are learned position embeddings where a model uses them.
6. In a decoder-only transformer of the Llama family, the linear layers carry no bias terms at all, and RMSNorm carries a scale but no shift. In GPT-2 the linear layers do carry biases and LayerNorm carries both scale and shift.
7. That difference is tiny in count. In GPT-2 small, all biases and all normalization parameters together are 121,344 out of 124,439,808 parameters, which is 0.098 per cent of the model.
8. Parameters are stored in the model’s state dictionary under dotted names. Real names from a Llama-family checkpoint look like this:

model.embed_tokens.weight            [128256, 4096]
model.layers.0.self_attn.q_proj.weight   [4096, 4096]
model.layers.0.self_attn.k_proj.weight   [1024, 4096]
model.layers.0.mlp.gate_proj.weight     [14336, 4096]
model.layers.0.input_layernorm.weight       [4096]
lm_head.weight                       [128256, 4096]

9. The bracketed pair is the tensor shape. The number of parameters in a tensor is the product of its dimensions.
10. Counting them in PyTorch is one line: sum(p.numel() for p in model.parameters()). Counting trainable ones only adds if p.requires_grad.
11. For a file on disk, safetensors stores a JSON header listing every tensor name, dtype and shape, so the count can be read without loading the model. For GGUF files, gguf-dump prints the same information.
12. Terminology note, and it is a standard one: in statistics a “parameter” is a quantity of a distribution, and in software a “parameter” is a function argument. Neither meaning applies here. The machine learning meaning is the one in this chapter.

WORDS47.1.6 remember these#

1. Parameter — one learned number in the model — one scalar element of a learnable tensor, updated by the optimizer, frozen at inference.
2. Weight — a number that multiplies something — a multiplicative parameter in a linear map, stored as an element of a weight matrix.
3. Bias — a number that is added on — an additive parameter applied after a linear map, one per output dimension.
4. Matrix — a rectangle of numbers — a rank-2 tensor with shape [out_features, in_features] for a linear layer.
5. Tensor — a box of numbers with a shape — an n-dimensional array with a dtype and a shape tuple.
6. State dictionary — the list of all the model’s numbers with names — an ordered mapping from parameter name to tensor, saved to disk.
7. Checkpoint — the saved file of a model — a serialized state dictionary, in formats such as safetensors, PyTorch .pt or GGUF.
8. Frozen — not changing any more — parameters with requires_grad=False, or any parameter during inference.

47.2 What 7B, 70B and 405B actually count#

PLAIN47.2.1 in simple words#

1. The B in 7B means billion. It is a count of parameters.
2. A 7B model contains about seven thousand million individual numbers.
3. A 405B model contains about four hundred and five thousand million of them.
4. Nothing else is being counted. Not files. Not facts. Not words known.
5. To get the total you add up the size of every matrix in the model.
6. Every matrix size is just rows times columns.
7. So the whole count is a long sum of simple multiplications.
8. The numbers are not round because the shapes are not round.
9. A model called “8B” often turns out to hold 8,030,261,248 parameters exactly.
10. The advertised name is rounded. The real count is exact and computable.
11. Once you know four or five shape numbers about a model, you can compute the total yourself in a few minutes.
12. That is what this section does, all the way through, for a real model.

PLAIN47.2.2 a picture in your head#

1. Imagine a large office block with 126 identical floors.
2. Every floor has exactly the same furniture plan: seven big filing cabinets.
3. Some cabinets are wide, some are narrow, but every floor gets the same set.
4. To count the drawers in the whole building you do not walk every floor.
5. You count the drawers on one floor, then multiply by 126.
6. Then you add the reception desk at the bottom and the roof office at the top, because those exist only once.
7. That is exactly how a transformer parameter count works.
8. The floors are the layers. The identical furniture is the same set of matrices repeated in every layer.
9. The reception desk is the embedding table. The roof office is the output layer. Both exist once, not once per floor.

Where this comparison breaks: real buildings sometimes have odd floors. In most transformer models every layer genuinely is the same shape, which is why the multiply-by-number-of-layers trick works exactly. But some newer designs break this on purpose: mixture-of-experts models have layers with many parallel expert blocks, and only some are used per token, so the total count and the count actually used per token stop being the same number.

PLAIN47.2.3 a worked example#

1. Let us count a real model completely: Llama 3.1 8B, released by Meta on 23 July 2024.
2. Five shape numbers describe it, and they are published:

3. Head size is hidden size divided by heads: 4096 / 32 = 128.
4. Now count one layer. Attention first, four matrices:

q_proj   4096 x 4096  = 16,777,216
k_proj   4096 x 1024  =  4,194,304
v_proj   4096 x 1024  =  4,194,304
o_proj   4096 x 4096  = 16,777,216
attention total per layer = 41,943,040

5. The k and v matrices are narrower because of grouped-query attention: only 8 key-value heads, so 8 x 128 = 1024 columns instead of 4096.
6. Now the feed-forward part, three matrices:

gate_proj   4096 x 14336 = 58,720,256
up_proj     4096 x 14336 = 58,720,256
down_proj  14336 x 4096  = 58,720,256
feed-forward total per layer = 176,160,768

7. Two normalization vectors of 4096 each add 8,192 more.
8. One layer therefore holds 41,943,040 + 176,160,768 + 8,192 = 218,112,000.
9. There are 32 layers: 32 x 218,112,000 = 6,979,584,000.
10. The embedding table is vocabulary times hidden size: 128,256 x 4,096 = 525,336,576.
11. The output layer is the same shape again: 525,336,576.
12. The final normalization adds 4,096.
13. Add them all: 6,979,584,000 + 525,336,576 + 525,336,576 + 4,096 = 8,030,261,248.
14. That is 8.03 billion. The model is sold as “8B”. The arithmetic closes.

PLAIN47.2.4 what is really happening inside#

1. Here is the whole 8B model drawn as a structure, with the parameter count written next to every piece.

       INPUT TOKEN IDS
              |
              v
  +--------------------------+
  | embedding table          |    525,336,576
  | 128,256 rows x 4,096     |
  +--------------------------+
              |
              v
  +==========================+  <-- repeated 32 times
  |  q_proj  16,777,216      |
  |  k_proj   4,194,304      |
  |  v_proj   4,194,304      |      attention
  |  o_proj  16,777,216      |      41,943,040
  |  ----------------------  |
  |  gate    58,720,256      |
  |  up      58,720,256      |      feed-forward
  |  down    58,720,256      |     176,160,768
  |  ----------------------  |
  |  2 norms      8,192      |
  |  layer total 218,112,000 |
  +==========================+
              |     x 32 layers = 6,979,584,000
              v
  +--------------------------+
  | final norm       4,096   |
  +--------------------------+
              |
              v
  +--------------------------+
  | output head              |    525,336,576
  | 128,256 rows x 4,096     |
  +--------------------------+
              |
              v
     ONE SCORE PER TOKEN         TOTAL 8,030,261,248

2. Look at where that total came from and a clear pattern appears.
3. The feed-forward matrices are the biggest single item, by a wide margin.
4. In the 8B model, feed-forward is 70 per cent of all parameters.
5. Attention is about 17 per cent.
6. The embedding and output tables together are about 13 per cent.
7. Normalization is a rounding error: three thousandths of one per cent.
8. Now scale the model up and watch the shares move.
9. In the 70B model, feed-forward rises to 80 per cent and embeddings fall to 3 per cent, because the vocabulary stays the same size while the layers grow.
10. In the 405B model, feed-forward is 81 per cent and embeddings are 1 per cent.
11. So the honest one-line answer to “where do the billions live” is: in the feed-forward blocks, repeated once per layer.
12. That matters practically. When people compress models, they attack the feed-forward matrices first, because that is where the mass is.
13. It also explains why small models feel embedding-heavy. In GPT-2 small, the embedding table alone is 25 per cent of the whole model.

TECHNICAL47.2.5 the engineer’s version#

1. The general formula for a decoder-only transformer with grouped-query attention, SwiGLU feed-forward and RMSNorm, with untied embeddings, is:

per_layer = 2*d*d                (q_proj and o_proj)
          + 2*d*n_kv*head_dim    (k_proj and v_proj)
          + 3*d*d_ff             (gate, up, down)
          + 2*d                  (two RMSNorm scales)

total = V*d                      (input embedding)
      + L*per_layer              (all layers)
      + d                        (final RMSNorm)
      + V*d                      (output head, if untied)

2. Here d is hidden size, d_ff feed-forward size, L layers, V vocabulary size, n_kv the number of key-value heads and head_dim the per-head width.
3. Applying it to the three published Llama 3.1 configurations gives exact totals that match the marketing names:

4. All three use vocabulary 128,256, head dimension 128, and 8 key-value heads. The 405B model has 128 query heads, which is why its k and v projections are only 16384 x 1024 while q and o are 16384 x 16384.
5. Where the parameters sit, as a percentage of the total:

6. Norm parameters are 0.0033 per cent for 8B, 0.0019 per cent for 70B and 0.0010 per cent for 405B, and are omitted from the table above for width.
7. Tied embeddings are an implementation choice, not a standard: the output projection reuses the transposed input embedding matrix, saving V x d parameters. GPT-2 ties them. Llama 3.1 8B, 70B and 405B do not. Several small models such as Gemma tie them, which is why their embedding share is larger than you would guess.
8. Verifying GPT-2 small the same way, including biases, gives exactly 124,439,808 parameters: 38,597,376 token embeddings, 786,432 learned position embeddings, 12 layers of 7,087,872 and a final LayerNorm of 1,536. The commonly quoted “117M” for GPT-2 small comes from an older count that excluded the position embeddings and some terms; 124M is the count that matches the released checkpoint.
9. For mixture-of-experts models the count splits in two. Report the total parameter count and the active parameter count separately, because only a few experts run per token. A model with 8 experts of which 2 run has roughly a quarter of its feed-forward parameters active at any moment. Quoting only the total for such a model is a marketing choice, not a technical one.
10. Tools: sum(p.numel() for p in model.parameters()) in PyTorch, model.num_parameters() in Hugging Face transformers, the safetensors index JSON header, and gguf-dump --no-tensors file.gguf for GGUF metadata.

WORDS47.2.6 remember these#

1. Hidden size — how wide the model is — d_model, the width of the residual stream carried between layers, commonly 4096 or 8192.
2. Feed-forward size — the width of the wide middle part — d_ff, usually between 2.7 and 4 times d_model in modern designs.
3. Layer — one repeat of the model’s standard block — one transformer block containing attention, feed-forward and two normalizations.
4. Vocabulary size — how many different tokens exist — the number of rows in the embedding matrix, for example 128,256 in Llama 3.1.
5. Grouped-query attention — sharing key and value work across heads — GQA, where n_kv is smaller than n_heads, shrinking k and v projections and the key-value cache.
6. Tied embeddings — reusing the input table as the output table — weight tying between embed_tokens and lm_head, saving V x d parameters.
7. Active parameters — the ones that actually run for a token — in a mixture-of-experts model, the subset routed to per token, as against the total stored.

47.3 Parameters versus the four things they get confused with#

PLAIN47.3.1 in simple words#

1. Five words get mixed up constantly. Here they are, one line each.
2. Parameters are the learned numbers inside the model, frozen after training.
3. Hyperparameters are the choices a human made before training started. They are not learned. Nobody trained them.
4. Activations are the temporary numbers created while the model runs on one input, and thrown away as soon as the answer is produced.
5. Context is the text you are giving the model right now. It is data.
6. Tokens are the small pieces that text is chopped into. Also data.
7. Now the sharp test. Which of these are in the model file on disk?
8. Only the parameters. The other four are not in the file.
9. Hyperparameters are recorded next to the file, in a small configuration note, so the loader knows what shape to build. But they were not learned.
10. Activations exist only for a moment while a request is being answered.
11. Context and tokens are whatever you happen to be asking about today.
12. So: parameters are what the model is. The rest is how it was built, what it is thinking about right now, and what you asked.

PLAIN47.3.2 a picture in your head#

1. Think of a bakery.
2. The recipe decisions are the hyperparameters: oven temperature, tin size, how long to bake. A person chose these before any baking started.
3. The baker’s learned skill is the parameters: the exact feel for how long to knead, built up over years and now fixed in the baker’s hands.
4. The ingredients on the counter today are the context: this flour, this water, this order for a birthday cake.
5. The individual scoops and spoonfuls are the tokens: the units the ingredients get measured into.
6. The half-made dough at each stage is the activations: sticky, temporary, real, and gone the moment the bread is out.
7. Now the useful part. Changing the oven temperature is changing a hyperparameter, and you have to re-train the baker to suit it. Giving new ingredients is changing the context, and needs no re-training at all.

Where this comparison breaks: a baker learns a little from every loaf. A deployed model does not. Its parameters are identical after your millionth message and after your first. Any apparent learning inside a conversation is the context being carried along, not the parameters changing.

PLAIN47.3.3 a worked example#

1. Here is one scenario with all five words used correctly in it.
2. A team trains a model. They choose 32 layers, hidden size 4096, learning rate 0.0003 and batch size 4 million tokens. Those are hyperparameters.
3. Training runs for weeks and produces 8,030,261,248 parameters, which are saved to a file of about 16 gigabytes.
4. A user types “Summarize this contract” and pastes 3,000 words of contract. That whole thing is the context.
5. The tokenizer chops that context into 4,182 tokens, each an integer.
6. The model runs. As those 4,182 tokens flow through 32 layers, it creates millions of temporary numbers at every layer. Those are activations, and they are discarded when the answer is finished.
7. Now the one-sentence version the reader asked for, all five in one line:
8. “The 8B model’s frozen parameters were produced by a training run whose hyperparameters fixed 32 layers and a learning rate of 0.0003; at inference the 4,182 tokens of the user’s context are turned into activations that flow through those parameters and are then thrown away.”
9. Read it twice. Every one of the five words is doing a different job.

PLAIN47.3.4 what is really happening inside#

1. Trace the lifetime of each of the five things, in order of when it exists.
2. Hyperparameters exist first, before there is a model at all. A human writes them into a configuration file.
3. The training program reads them and builds an empty model of that shape, filled with random numbers.
4. Training begins. The parameters change, millions of times, over weeks.
5. Training ends. The parameters are written to disk and never change again.
6. Later, somebody loads the file. Now the parameters are in memory.
7. A user sends text. That is the context. The tokenizer converts it to tokens.
8. The tokens flow in. At every layer the model computes new temporary numbers. Those are activations.
9. The activations are what carries the meaning of your specific question. The parameters are what does the transforming.
10. When the reply is finished, the activations are freed. The parameters are still sitting there, byte for byte identical.
11. This is why inference needs more memory than the model file size. The activations, and the cached keys and values for the conversation so far, need room too.
12. And it is why a longer conversation costs more memory but does not make the model bigger. The model is exactly as big as it was.

TECHNICAL47.3.5 the engineer’s version#

1. Hyperparameters split into two families that people wrongly lump together. Architectural hyperparameters fix the shape and therefore fix the parameter count: number of layers, hidden size, feed-forward size, head count, vocabulary size, maximum position count. Optimization hyperparameters do not change the count at all: learning rate, warmup steps, weight decay, batch size, dropout rate, gradient clipping threshold, number of epochs.
2. Sampling settings supplied per request, such as temperature, top-p, top-k and repetition penalty, are sometimes loosely called hyperparameters. They are inference settings. They change decoding, not the model.
3. Activations are the intermediate tensors of the forward pass. During inference with no gradient tracking, most are freed as soon as the next layer consumes them, so peak activation memory is a few hundred megabytes for a single sequence, not gigabytes.
4. During training the picture inverts. Activations must be retained for the backward pass. Using the accounting from the 2022 NVIDIA paper Reducing Activation Recomputation in Large Transformer Models, activation memory per layer per sample is about s*b*h*(34 + 5*a*s/h) bytes in 16-bit, where s is sequence length, b batch, h hidden size and a head count.
5. For a 7B-class model at sequence length 2048 and batch 1 that is about 912 MiB per layer, so about 28.5 GiB across 32 layers for a single sample, before any recomputation is applied. Gradient checkpointing trades compute for memory to cut that sharply.
6. The key-value cache is a special, persistent kind of activation. It is the only activation deliberately kept across decoding steps, and it grows linearly with context length. It is covered with numbers in section 47.4.
7. Context is data on the request path. It has no gradient, no persistence and no effect on any stored parameter. This is the technical reason a model cannot “learn from your chat” unless a separate fine-tuning job is run.
8. Tokens are integers in the range 0 to V-1. They index rows of the embedding matrix. They are not parameters; the rows they select are.
9. Observing them: config.json and generation_config.json hold hyperparameters, nvidia-smi and torch.cuda.max_memory_allocated() show activation and cache growth, and any tokenizer’s encode returns the tokens.

WORDS47.3.6 remember these#

1. Hyperparameter — a choice made before training — a configuration value not updated by gradient descent, either architectural or optimization-related.
2. Activation — a temporary number inside the model while it runs — an intermediate tensor of the forward pass, freed unless retained for backward.
3. Context — the text being processed right now — the input token sequence supplied per request, including any system prompt and history.
4. Key-value cache — the model’s notes on the conversation so far — cached per-layer key and value tensors reused across decoding steps.
5. Inference — using the model — a forward pass with no gradient computation and no parameter update.
6. Forward pass — pushing data through the model once — computing outputs layer by layer from inputs.
7. Gradient checkpointing — recomputing instead of storing — discarding activations during forward and recomputing them during backward to save memory at the cost of extra compute.

47.4 Parameters, memory and precision#

PLAIN47.4.1 in simple words#

1. Every parameter has to be stored somewhere, and storage takes space.
2. So the memory a model needs is a simple multiplication.
3. Memory = number of parameters x bytes used per parameter.
4. That is the whole rule. There is no magic in it.
5. How many bytes per parameter? That is your choice, and it is a trade.
6. Four bytes stores a number very accurately. Two bytes stores it less accurately but takes half the space.
7. One byte is less accurate still. Half a byte, which means four bits, is cruder again.
8. A 7 billion parameter model at two bytes each needs about 14 gigabytes.
9. The same model at half a byte each needs about 3.5 gigabytes.
10. Same model. Same parameters. Different amount of detail kept per number.
11. But the weights are not the only thing in memory when the model runs.
12. There is also the model’s running notes on your conversation, the temporary numbers, and the software’s own overhead.
13. And training is far worse again, because training keeps several extra copies of every parameter. We will work that out too.

PLAIN47.4.2 a picture in your head#

1. Think of writing down the price of something.
2. You could write 12.345678. That is very precise and takes many characters.
3. You could write 12.35. Less precise, fewer characters.
4. You could write 12. Cruder still, fewest characters.
5. If you have to write down seven billion prices, the choice matters enormously for how much paper you need.
6. And for most purposes 12.35 is good enough. Nobody notices the missing digits when you are adding up a shopping basket.
7. That is exactly the precision trade in a model. Fewer digits per number, much less memory, slightly less exact answers.
8. Now the training part. To train, you do not only need the current prices.
9. You also need, for every price, a note of which direction to move it, and two running averages of how it has been moving lately.
10. So training needs about four to five sheets of paper for every one sheet that using the finished model needs.

Where this comparison breaks: numbers in a computer are not stored as decimal digits. They are stored in a floating point format with a sign, an exponent and a fraction, and the loss of accuracy is not a clean chopping of digits. It is a coarser grid of representable values, and the grid spacing changes with magnitude.

PLAIN47.4.3 a worked example#

1. Take the same models at four different precisions. Here is the arithmetic for weights only, using 1 GiB = 1,073,741,824 bytes.
2. 7 billion x 4 bytes = 28,000,000,000 bytes = 26.1 GiB.
3. 7 billion x 2 bytes = 14,000,000,000 bytes = 13.0 GiB.
4. 7 billion x 1 byte = 7,000,000,000 bytes = 6.5 GiB.
5. 7 billion x 0.5 bytes = 3,500,000,000 bytes = 3.3 GiB.
6. Repeat for the other sizes and the full table falls out:

7. Sanity check against reality: the published fp16 file for Llama 3.1 70B is about 141 GB in decimal gigabytes, which is 131 GiB. The arithmetic matches the real file.
8. Now add what inference needs beyond the weights.
9. The key-value cache for Llama 3.1 70B is 320 KiB per token. At 8,000 tokens that is 2.5 GiB. At 32,000 tokens it is 10 GiB.
10. Temporary activations for one sequence are roughly 0.5 to 1 GiB.
11. The framework’s own overhead, buffers and fragmentation is another 1 to 2 GiB in practice.
12. So a realistic total for 70B at fp16 with an 8,000 token context is 131.4 + 2.5 + 1.0 + 2.0 = about 137 GiB, not 131.

PLAIN47.4.4 what is really happening inside#

1. Training memory is a different story, and much bigger, for a simple reason.
2. Training must keep, for every single parameter, four extra things.
3. First, the parameter itself, usually kept twice: a fast low-precision copy used for the maths and an accurate master copy used for the update.
4. Second, the gradient, which is one number per parameter saying which way to nudge it.
5. Third and fourth, the optimizer’s two running averages. The standard optimizer, called Adam, keeps a running average of recent gradients and a running average of recent squared gradients.
6. That is what “Adam’s two moments” means: two extra numbers per parameter.
7. Count the bytes for the common mixed-precision recipe:

16-bit working weights          2 bytes
32-bit master weights           4 bytes
gradients                       4 bytes
Adam first moment  (m)          4 bytes
Adam second moment (v)          4 bytes
                              ----------
total                          18 bytes per parameter

8. For a 7B model: 7,000,000,000 x 18 = 126,000,000,000 bytes = 117 GiB.
9. That is before a single activation is stored.
10. Add training activations, which for a 7B-class model at sequence length 2048 come to roughly 28 GiB per sample without recomputation.
11. So training a 7B model is a 150 GiB-plus problem, not a 14 GiB problem.
12. That is why a model you can happily run on one 24 GiB graphics card cannot be fully fine-tuned on it. Using needs weights. Training needs weights plus four more copies plus history.
13. It is also why techniques that train only a small extra set of parameters, which Chapter 50 covers, exist at all.

TECHNICAL47.4.5 the engineer’s version#

1. The precision formats, with their real layouts:

2. bfloat16 was introduced by Google for its Tensor Processing Units and keeps fp32’s 8-bit exponent, so it has the same dynamic range as fp32 with far fewer fraction bits. That is why bf16 training rarely needs loss scaling while fp16 training usually does.
3. fp16 has a maximum finite value of 65504 and underflows below about 6e-8, which is why gradients in fp16 are scaled up before the backward pass and scaled down afterwards.
4. fp8 in the E4M3 and E5M2 variants was standardized across NVIDIA, Arm and Intel in a 2022 joint proposal, and is supported in hardware from NVIDIA’s Hopper generation, announced March 2022. NVIDIA’s Blackwell generation, announced 18 March 2024, added hardware support for 4-bit floating point.
5. Key-value cache size per token is exact and worth memorizing:

kv_bytes_per_token = 2 * L * n_kv * head_dim * bytes_per_element

6. The leading 2 is for keys and values. Real figures at fp16:

7. That last row is the point of grouped-query attention. Cutting key-value heads from 64 to 8 cuts the cache by a factor of 8, which is what makes long contexts affordable at all.
8. Full-fine-tuning memory in bytes per parameter, by optimizer:

9. Adam was published by Diederik Kingma and Jimmy Ba in 2014 as Adam: A Method for Stochastic Optimization; AdamW, the decoupled weight decay variant used by almost every modern language model, came from Ilya Loshchilov and Frank Hutter in 2017. Both keep two moment estimates per parameter, which is the source of the 8 bytes.
10. ZeRO, from the 2019 Microsoft paper ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, shards optimizer states, gradients and parameters across data-parallel ranks in three stages, so the per-device cost falls roughly by the number of devices. It does not reduce the total.
11. Practical observation commands: nvidia-smi --query-gpu=memory.used --format=csv, torch.cuda.memory_summary(), and for serving, vLLM’s reported KV cache blocks and --gpu-memory-utilization setting.

WORDS47.4.6 remember these#

1. Precision — how many digits are kept per number — the numeric format’s bit width and its split between exponent and fraction.
2. fp32 — the accurate, expensive format — IEEE 754 single precision, 4 bytes, 24 bits of significand including the implicit bit.
3. bf16 — the training-friendly 16-bit format — brain floating point, fp32’s exponent range with 8 bits of significand.
4. Key-value cache — the model’s stored notes on tokens seen so far — cached K and V tensors per layer, sized 2 x L x n_kv x head_dim per token.
5. Optimizer state — the extra bookkeeping training needs — Adam’s first and second moment estimates, one of each per parameter.
6. Mixed precision — computing in 16-bit, updating in 32-bit — a training recipe with a low-precision working copy and an fp32 master copy.
7. GiB versus GB — two different gigabytes — 2^30 = 1,073,741,824 bytes versus 10^9 bytes; file sizes are usually quoted in the decimal form and memory in the binary form, and confusing them costs you 7 per cent.

47.5 Quantization#

PLAIN47.5.1 in simple words#

1. Quantization means storing each of the model’s numbers using fewer bits.
2. That is the whole idea in one line. Fewer bits per number, smaller model.
3. The trained numbers are typically stored in 16 bits each.
4. Quantization rewrites them into 8 bits, or 4 bits, or sometimes fewer.
5. Nothing about the model’s shape changes. It still has 70 billion parameters.
6. Each of those 70 billion is just written down more roughly.
7. To do it you need a trick, because a 4-bit slot holds only 16 different values, and the real weights are spread over a wide range.
8. The trick is to store a small group of weights together with one shared scaling number.
9. The 4-bit value says “which of my 16 steps”, and the shared scale says “and here is how big a step is”.
10. Multiply them back together at load time or on the fly, and you get an approximation of the original number.
11. The approximation is not exact. The question is whether the error matters.
12. Mostly, at 8 bits and 5 bits, it barely does. At 4 bits it is small. Below that it starts to hurt in ways you can measure.

PLAIN47.5.2 a picture in your head#

1. Imagine a shop that must label 10,000 items with prices.
2. The prices range from 1 rupee to 3,000 rupees.
3. The labels only have room for a single digit, 0 to 9.
4. That looks impossible, until you split the shop into departments.
5. In the pencil department, prices run 1 to 10 rupees, so you write on the department sign: “each step on the label = 1 rupee”.
6. In the furniture department, prices run 1,000 to 3,000, so that sign says “each step on the label = 200 rupees”.
7. Now a label reading 7 means 7 rupees in one department and 1,400 in the other, and one digit is enough everywhere.
8. The department sign is the scale. The department is the group.
9. Smaller departments mean more accurate prices but more signs to store.
10. That is the entire design space of quantization: how small the groups are, and how much extra you pay to store the signs.

Where this comparison breaks: real quantization also needs a zero point when the values are not centred on zero, so the label’s 0 does not have to mean a value of 0. And unlike a shop, a badly chosen department boundary does not just annoy a customer; it puts an outlier weight far outside the range and distorts every weight in the group with it.

PLAIN47.5.3 a worked example#

1. Take eight real-looking weights and quantize them by hand.
2. The values: 0.42, -1.13, 0.05, 2.31, -0.77, 1.02, -2.90, 0.66.
3. Symmetric 8-bit first. Find the largest absolute value: 2.90.
4. Scale = 2.90 / 127 = 0.022835.
5. Divide each value by the scale and round:

value    /scale    rounded int8
 0.42     18.39         18
-1.13    -49.49        -49
 0.05      2.19          2
 2.31    101.16        101
-0.77    -33.72        -34
 1.02     44.67         45
-2.90   -127.00       -127
 0.66     28.90         29

6. To use them, multiply back by the scale. 18 x 0.022835 = 0.4110.
7. Original 0.42, recovered 0.4110. Error 0.0090.
8. Across all eight, the largest error is 0.0111. Small.
9. Now 4-bit, asymmetric, using minimum and maximum instead of absolute maximum.
10. Minimum -2.90, maximum 2.31. Scale = (2.31 - (-2.90)) / 15 = 0.347333.
11. Zero point = round(2.90 / 0.347333) = 8. So the stored code 8 means 0.
12. Quantized codes: 9, 5, 8, 15, 6, 11, 0, 10.
13. Recovered values: 0.3473, -1.0420, 0.0000, 2.4313, -0.6947, 1.0420, -2.7787, 0.6947.
14. Largest error now 0.1213, about eleven times worse than 8-bit. That is the price of four bits instead of eight.
15. Note the storage cost of the scale itself. If one 16-bit scale and one 16-bit zero point cover a group of 64 weights, the real cost is 4 + 32/64 = 4.5 bits per weight, not 4.

PLAIN47.5.4 what is really happening inside#

1. There are three levels of granularity, and they are the main design choice.
2. Per-tensor: one scale for an entire matrix. Cheapest to store, worst accuracy, because one outlier anywhere ruins the scale for everything.
3. Per-channel: one scale per row or per column. Much better, because a badly behaved output channel no longer poisons its neighbours.
4. Per-group: one scale for every fixed run of weights, typically 32, 64, 128 or 256 of them. Best accuracy, most scales to store.
5. Then there is the question of when you quantize.
6. Post-training quantization takes a finished model and compresses it. Fast, needs no training run, sometimes needs a few hundred sample texts to calibrate the scales.
7. Quantization-aware training simulates the rounding during training, so the model learns weights that survive rounding well. Much more expensive, better results at very low bit widths.
8. Finally: not everything gets squeezed equally.
9. The embedding table and the output layer are often kept at higher precision, because errors there hit every token directly and there is no later layer to absorb the damage.
10. Normalization scales, which are a tiny fraction of parameters, are almost always left at 16 or 32 bits.
11. Some formats keep specific matrices, such as the attention value projection and the feed-forward down projection, at a higher bit width because measurements showed those layers are the most sensitive.
12. And the key-value cache is a separate decision. You can run 4-bit weights with a 16-bit cache, or quantize the cache too.

TECHNICAL47.5.5 the engineer’s version#

1. The affine quantization map, which is the standard, is:

q = clamp(round(x / s) + z, qmin, qmax)
x_hat = (q - z) * s

2. Symmetric quantization fixes z = 0 and sets s = max|x| / qmax. Asymmetric quantization derives s from the range and z from the offset. Symmetric is cheaper at run time because the zero point drops out of the arithmetic.
3. The named methods, with correct attribution:

4. LLM.int8(), by Tim Dettmers and colleagues in 2022, observed that a small number of feature dimensions carry extreme magnitudes and must stay in 16 bits, while the rest go to int8. This is mixed-precision decomposition.
5. GPTQ, by Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh, posted October 2022 and published at ICLR 2023, quantizes weights column by column and uses approximate second-order information from a small calibration set to adjust the not-yet-quantized weights to compensate for the error just introduced.
6. AWQ, Activation-aware Weight Quantization, by Ji Lin and colleagues at MIT, posted June 2023 and awarded Best Paper at MLSys 2024, observes that protecting roughly 1 per cent of weights, chosen by activation magnitude rather than weight magnitude, recovers most of the loss, and implements the protection by per-channel scaling rather than by keeping mixed precision.
7. NF4, NormalFloat4, comes from the QLoRA paper by Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer, May 2023. Its 16 levels are placed at the quantiles of a normal distribution rather than evenly, because trained weights are approximately normally distributed. QLoRA also applies double quantization, quantizing the scales themselves, saving about 0.37 bits per parameter.
8. GGUF is the file format used by llama.cpp, introduced on 21 August 2023 as the successor to GGML, GGMF and GGJT. Its k-quant types use a super-block of 256 weights subdivided into blocks of 32. Exact sizes, read from the ggml header:

9. The suffixes S and M mean small and medium mixes. Q4_K_M keeps the attention value projections and the feed-forward down projections at Q6_K in part of the network, so its measured average is about 4.8 bits per weight rather than 4.5. Checking against real published files: Llama 3.1 8B at Q4_K_M is 4.9 GB, and 8.03e9 x 4.83 / 8 = 4.85e9 bytes. The published 405B Q4_K_M file is 243 GB, and 405.85e9 x 4.79 / 8 = 243e9. The arithmetic closes both ways.
10. Quality cost, measured. From a January 2026 evaluation of llama.cpp quantization on Llama-3.1-8B-Instruct, WikiText-2 perplexity, lower better:

11. Read that table honestly. Q8_0 costs 0.01 perplexity, which is noise. Q4_K_M costs 0.24, about 3.3 per cent relative, for a 69 per cent size cut. Q3_K_S costs 1.64, about 22 per cent relative, and that is a real degradation you will notice.
12. The same study reported that on aggregate downstream scores across GSM8K, HellaSwag, IFEval, MMLU and TruthfulQA, several quantized variants scored within noise of the fp16 baseline, and Q5_0 scored marginally above it. That last result is a reminder that benchmark noise at these margins is larger than the quantization effect, not evidence that quantizing helps.
13. Now the laptop arithmetic the question asks for. Assume about 75 per cent of installed memory is usable for the model, the rest going to the operating system and everything else.

Machine with 64 GiB, about 48 GiB usable:
  70B at fp16   131.4 GiB weights  -> does not fit
  70B at int8    65.7 GiB weights  -> does not fit
  70B at Q4_K_M  39.7 GiB weights
                + 2.5 GiB KV at 8k
                + 2.5 GiB other    -> 44.7 GiB, fits

Machine with 16 GiB, about 12 GiB usable:
  8B at fp16     15.0 GiB weights  -> does not fit
  8B at int8      7.5 GiB weights
                + 1.0 GiB KV at 8k
                + 1.3 GiB other    -> 9.8 GiB, fits
  8B at Q4_K_M    4.5 GiB weights  -> 6.8 GiB total, fits easily
  70B at Q4_K_M  39.7 GiB weights  -> does not fit

14. So the honest answer to “why can a 70B model run on a laptop at 4-bit but not at 16-bit” is: 131.4 GiB does not fit in 48 GiB and 39.7 GiB does, and that is the entire reason. Note the 64 GiB case only works with a modest context. Raise the context to 32,000 tokens and the cache alone adds 10 GiB, taking the total to about 52 GiB, which no longer fits.
15. On Apple silicon the calculation uses unified memory, and the default cap on how much the GPU may claim is around 65 to 75 per cent of installed RAM depending on machine and macOS version, which is where the 75 per cent rule of thumb above comes from.
16. Speed, separately from memory: at batch size 1, generation is memory bandwidth bound, so halving the bytes per weight roughly doubles tokens per second. That is often a bigger practical win than the memory saving.
17. Tools: llama-quantize in llama.cpp, AutoGPTQ and GPTQModel, autoawq, bitsandbytes for NF4 and int8, llm-compressor for vLLM-targeted formats, and optimum-quanto in the Hugging Face stack.

WORDS47.5.6 remember these#

1. Quantization — storing each number in fewer bits — mapping high-precision weights onto a small integer grid with a scale and optional zero point.
2. Scale — how big one step of the integer grid is — the multiplier s in x_hat = (q - z) * s.
3. Zero point — where the integer grid’s zero sits — the offset z that lets an asymmetric range be represented.
4. Group size — how many weights share one scale — commonly 32, 64 or 128; smaller means more accurate and more metadata.
5. Post-training quantization — squeezing a finished model — PTQ, applied to trained weights with at most a small calibration set.
6. Quantization-aware training — training with the rounding simulated — QAT, with fake quantization in the forward pass and straight-through gradients.
7. Perplexity — how surprised the model is by real text — the exponential of the mean negative log-likelihood per token; lower is better, and only comparable between models sharing a tokenizer.
8. Outlier channel — a feature dimension with extreme values — a dimension whose magnitude forces a bad scale for everything grouped with it.

47.6 Why capability does not scale linearly with parameter count#

PLAIN47.6.1 in simple words#

1. Bigger models are, on average, better. That much is true.
2. But parameter count is a weak way to guess how good a model is.
3. A model is the product of at least five things, not one.
4. How many parameters it has. How much text it was trained on. How good that text was. How much computing time was spent. What was done to it afterwards.
5. Get any one of those badly wrong and a huge model performs poorly.
6. Get them all right and a small model can beat a much larger one.
7. This is not a rare exception. It happens constantly.
8. In 2022 a 70 billion parameter model called Chinchilla beat a 280 billion parameter model called Gopher, using the same computing budget.
9. It won by being smaller and reading four times as much text.
10. That result changed how the whole field trains models.
11. So when a company says its model has more parameters than a rival’s, that is a fact about the file size, not a claim about usefulness that you should accept without evidence.
12. The honest position: parameter count sets a ceiling. It does not tell you how close to that ceiling the model got.

PLAIN47.6.2 a picture in your head#

1. Think of a student preparing for an exam.
2. Parameter count is how much the student could remember if they studied. Call it brain capacity.
3. Training data is how many books they read.
4. Data quality is whether the books were good ones or nonsense from the internet.
5. Training compute is how many hours they actually spent studying.
6. The stages after training are the coaching: practice papers, feedback on answers, learning how to lay out a solution clearly.
7. Now: a student with enormous capacity who read three books and never practised will fail.
8. A student with ordinary capacity who read widely, studied hard and got good coaching will do well.
9. Everyone knows this about students. The same thing is true about models, and the industry spent years talking as though only capacity mattered.

Where this comparison breaks: a student’s capacity is fixed and unknown; a model’s is chosen and exactly known. And a model has no motivation, no sleep and no forgetting, so the analogy tells you nothing about the training dynamics. It only tells you that one input out of five does not determine the output.

PLAIN47.6.3 a worked example#

1. Take the Chinchilla experiment, published by Jordan Hoffmann and colleagues at DeepMind in March 2022, in Training Compute-Optimal Large Language Models.
2. Gopher: 280 billion parameters, trained on 300 billion tokens.
3. Chinchilla: 70 billion parameters, trained on 1.4 trillion tokens.
4. Both used about the same total computing budget. That is the key control.
5. Chinchilla is four times smaller and read four and a half times more.
6. On MMLU, a broad multiple-choice benchmark, Chinchilla scored 67.5 per cent, more than 7 percentage points above Gopher.
7. It also beat GPT-3 at 175 billion parameters, Jurassic-1 at 178 billion and Megatron-Turing NLG at 530 billion.
8. A 70B model beat a 530B model. That is a factor of seven and a half in parameter count, going the wrong way for the bigger model.
9. The paper’s rule of thumb: for a fixed compute budget, model size and training tokens should grow together, roughly 20 tokens per parameter.
10. By that rule, GPT-3 at 175B should have been trained on about 3.5 trillion tokens. It was trained on 300 billion. It was undertrained by roughly a factor of ten.
11. So the entire generation of models before 2022 was too big for the amount of text it had read.

PLAIN47.6.4 what is really happening inside#

1. Training reduces a single measured quantity: the average surprise the model feels when it sees real text.
2. That quantity falls smoothly and predictably as you add parameters, add data or add compute. Those smooth curves are called scaling laws.
3. A scaling law is a fitted curve, not a law of nature. It describes what has been measured over the ranges measured.
4. The curves are also very shallow. To halve the error you need far more than double the size.
5. And low surprise on text is not the same thing as being useful to a person.
6. That gap is closed by the stages after pre-training: supervised fine-tuning on demonstrations, and preference training on human comparisons.
7. Those stages change very few things about the model’s knowledge and a great deal about its behaviour.
8. A raw pre-trained model of 70 billion parameters is often unusable as an assistant. The same model after post-training is helpful. Same parameters, different usefulness.
9. Data quality does something similar. Training on carefully filtered, deduplicated, high-quality text produces much better models per parameter than training on raw web scrape.
10. So the causal chain from parameter count to usefulness runs through at least four other factors, each of which can dominate.

TECHNICAL47.6.5 the engineer’s version#

1. Separate the three kinds of claim, as the reader asked.
2. Established fact. Loss falls as a power law in parameters, data and compute over several orders of magnitude. Kaplan and colleagues at OpenAI established this in January 2020 in Scaling Laws for Neural Language Models.
3. Established fact. Kaplan’s recommended allocation was corrected by Hoffmann and colleagues in March 2022. Under a fixed compute budget, the compute-optimal ratio is roughly 20 training tokens per parameter, and pre-2022 models were badly undertrained.
4. Established fact. The Chinchilla scaling coefficients were partially disputed. Epoch AI published a replication attempt in 2024 finding that the paper’s third estimation approach had fitting problems, though the broad conclusion, that data and parameters should scale together, survived.
5. Established fact, and important. Compute-optimal is not deployment-optimal. If a model will serve billions of requests, it is rational to train a smaller model far past its Chinchilla point, because inference cost is paid forever and training cost is paid once. Llama 3 8B was trained on about 15 trillion tokens, which is roughly 1,875 tokens per parameter, nearly 100 times the Chinchilla ratio, and it is a much better 8B model for it.
6. Established fact. Data curation matters enormously per parameter. Microsoft’s phi line, starting with the June 2023 paper Textbooks Are All You Need, showed that filtered and synthetic textbook-style data lets very small models reach scores that previously needed models many times larger, though critics noted benchmark contamination risk in such setups.
7. Established fact. Mistral 7B, released 27 September 2023, outperformed Llama 2 13B on all benchmarks its authors reported. A 7B beat a 13B. Neither architecture nor data was held constant, which is exactly the point: parameter count alone did not decide it.
8. Active research. How much of frontier capability comes from post-training as against pre-training scale; whether reasoning-focused post-training and test-time compute change the scaling picture; how to measure data quality before spending the compute.
9. Marketing claim. Any headline of the form “our model has N parameters, therefore it is better”. Also any comparison that quotes total parameters for a mixture-of-experts model against dense parameters for a rival without saying so.
10. Where experts disagree. Some argue continued scaling of pre-training is still the main driver of progress; others argue returns from pre-training scale have flattened and that data quality, post-training and inference-time compute now dominate. Both camps include serious researchers, and the public evidence does not settle it, partly because frontier training details are no longer published.
11. Practical consequence for an engineer: never select a model by parameter count. Select by measured performance on your own task, then by cost and latency. Parameter count is only useful for predicting memory and speed, which it predicts extremely well.

WORDS47.6.6 remember these#

1. Scaling law — a curve saying how much better a bigger model gets — a fitted power law relating loss to parameters, data or compute.
2. Compute-optimal — the best size for a fixed training budget — the parameter-and-token split minimizing loss at fixed FLOPs, about 20 tokens per parameter per Chinchilla.
3. Undertrained — too big for the text it read — a model whose token count is far below its compute-optimal allocation.
4. Overtrained (deliberately) — trained far past compute-optimal on purpose — a smaller model given many more tokens to reduce lifetime inference cost.
5. Pre-training — the long first phase — self-supervised next-token prediction on a very large corpus.
6. Post-training — everything done afterwards — supervised fine-tuning, preference optimization and safety training, covered in Chapter 50.
7. MMLU — a broad general-knowledge exam for models — Massive Multitask Language Understanding, 57 subjects of multiple-choice questions, released 2020.

47.7 Tokens: why text is not fed in as words or characters#

PLAIN47.7.1 in simple words#

1. A model works on numbers, so text must become numbers first.
2. The obvious plan is: one number per word. It fails, for four reasons.
3. First, there is no fixed list of all words. New ones appear every day.
4. Second, typos. “recieve” is not in any word list, so it becomes an unknown.
5. Third, many languages do not put spaces between words at all.
6. Fourth, a word list big enough to be useful is enormous, and the model would need one row of numbers per word, which is very expensive.
7. The opposite plan is: one number per letter. That fails too.
8. A page of text becomes thousands of letters, so sequences get very long, and long sequences are expensive.
9. Worse, the model must then learn spelling from scratch before it can learn anything about meaning.
10. So neither extreme works. The answer is in between: chop text into pieces that are usually bigger than a letter and often smaller than a word.
11. Those pieces are called tokens or subwords.
12. Common words become one token. Rare words become several pieces.
13. A word nobody has ever seen still gets encoded, because in the worst case it falls back to very small pieces.
14. Nothing is ever unknown. That is the property that makes it work.

PLAIN47.7.2 a picture in your head#

1. Think about how a printer’s typecase used to work.
2. One option: cut a separate metal block for every word in the language. Millions of blocks, and a new one every time somebody coins a word.
3. Another option: keep only 26 letter blocks and build everything letter by letter. Very few blocks, but setting one page takes forever.
4. What real printers did was in between. They kept letters, and they also kept ligatures: single blocks for common combinations like “fi” and “ffl”.
5. Frequent combinations got their own block because it saved work.
6. Rare combinations were built up from letters.
7. That is exactly what subword tokenization does, chosen automatically by counting which combinations are frequent in a big pile of text.

Where this comparison breaks: a printer’s ligatures were chosen by human craftsmen for how they looked. A tokenizer’s pieces are chosen by a counting algorithm for how often they occur, and many of them look strange to a human eye. There is also no requirement that a token be a meaningful unit. Tokens regularly cut straight through the middle of a morpheme.

PLAIN47.7.3 a worked example#

1. Here is byte-pair encoding worked completely, on a tiny corpus.
2. The corpus is four words with counts: low appears 5 times, lower 2 times, newest 6 times, widest 3 times.
3. Start by splitting every word into characters, with an underscore marking the end of a word.
4. The starting symbol set is 11 symbols: _ d e i l n o r s t w.
5. Now count every adjacent pair, weighted by word frequency, and merge the most frequent pair. Repeat.

step 1: top pairs  es=9  st=9  t_=9  we=8  lo=7  ow=7
        merge e+s -> es
  l o w _        (5)
  l o w e r _    (2)
  n e w es t _   (6)
  w i d es t _   (3)

step 2: top pairs  est=9  t_=9  lo=7  ow=7  ne=6  ew=6
        merge es+t -> est
  n e w est _    (6)
  w i d est _    (3)

step 3: merge est+_ -> est_       (count 9)
  n e w est_     (6)
  w i d est_     (3)

step 4: merge l+o -> lo           (count 7)
step 5: merge lo+w -> low         (count 7)
  low _          (5)
  low e r _      (2)

step 6: merge n+e -> ne           (count 6)
step 7: merge ne+w -> new         (count 6)
step 8: merge new+est_ -> newest_ (count 6)
step 9: merge low+_ -> low_       (count 5)
step 10: merge w+i -> wi          (count 3)

6. After 10 merges the words look like this: low_ (5), low e r _ (2), newest_ (6), wi d est_ (3).
7. The learned merge list, in order, is: es, est, est_, lo, low, ne, new, newest_, low_, wi.
8. The vocabulary is now 11 starting symbols + 10 merges = 21 tokens.
9. Notice what happened. “newest” became a single token because it was frequent.
10. “lower” is still in pieces, because it appeared only twice.
11. “est_” became a token that means, roughly, the superlative ending. Nobody told the algorithm about grammar. It fell out of counting.
12. Real training does this 30,000 to 200,000 times instead of 10 times, on billions of words instead of 16.

PLAIN47.7.4 what is really happening inside#

1. Training a tokenizer and using one are two different programs.
2. Training: count pairs, merge the most frequent, repeat until the vocabulary reaches the target size. Save the ordered merge list and the vocabulary.
3. Using: split the incoming text into characters, then apply the saved merges in exactly the order they were learned. Same order, every time.
4. That ordering is what makes the process deterministic. The same text always gives the same tokens.
5. Then each final piece is looked up in the vocabulary to get its integer id.
6. A crucial detail: modern tokenizers do not start from characters. They start from bytes.
7. There are exactly 256 possible byte values, so every possible input, in any language, in any encoding, including broken text, can always be represented.
8. That is why you never see an “unknown token” error from a modern model.
9. The cost of the byte fallback is that unusual scripts get split into individual bytes, and one character can become three or four tokens.
10. There is also a pre-tokenization step. Before any merging, the text is split by a fixed pattern, usually keeping leading spaces attached to words and splitting runs of digits into groups.
11. That pattern is why " the" with a leading space is a different token from “the” without one, and both exist in the vocabulary.

TECHNICAL47.7.5 the engineer’s version#

1. The four families you need to be able to name and tell apart:

2. Byte-pair encoding was invented as a data compression algorithm by Philip Gage, published in The C Users Journal in February 1994. Rico Sennrich, Barry Haddow and Alexandra Birch adapted it to machine translation in Neural Machine Translation of Rare Words with Subword Units, posted 2015 and published at ACL 2016. GPT-2 in 2019 moved BPE to operate on raw bytes, giving the 256-symbol base alphabet and eliminating unknown tokens.
3. WordPiece came from Mike Schuster and Kaisuke Nakajima at Google in the 2012 ICASSP paper Japanese and Korean Voice Search. It merges the pair that most increases the likelihood of the training data under a unigram language model, rather than the most frequent pair. Concretely it maximizes count(xy) / (count(x) * count(y)), so a pair of two rare symbols can beat a pair of two common ones. BERT, released by Google in October 2018, uses WordPiece, and marks word continuations with a double hash prefix.
4. Unigram language model tokenization was introduced by Taku Kudo in the 2018 ACL paper Subword Regularization. It works in the opposite direction: start with a large candidate vocabulary, score every candidate by how much removing it would hurt the corpus likelihood, and iteratively prune the worst until the target size is reached. Because it keeps a probability per token, it can sample alternative segmentations of the same string, which is used as a data augmentation technique.
5. SentencePiece is a library, not an algorithm, from Taku Kudo and John Richardson, EMNLP 2018 system demonstrations. It implements both BPE and Unigram, and its real contribution is treating input as a raw stream with no language-specific pre-tokenization, encoding the space character itself as a visible symbol so that detokenization is exactly reversible. That is what makes it work for Japanese, Chinese and Thai, which do not delimit words with spaces. T5, Llama 1 and Llama 2 use SentencePiece; Llama 3 moved to a tiktoken-style byte-level BPE with a 128,000-entry vocabulary.
6. Real vocabulary sizes, for scale:

7. Larger vocabularies mean fewer tokens per document, which means cheaper attention and shorter sequences, but a larger embedding and output matrix. Going from 50k to 200k vocabulary at hidden size 4096 costs about 1.2 billion extra parameters across the two tables.
8. Special tokens are added outside the learned merges: sequence start, sequence end, padding, and chat role markers such as a header start marker. These are reserved ids that the merge process can never produce from text, which is a deliberate security property: user text cannot forge a role boundary.
9. Tools: tiktoken for the OpenAI encodings, tokenizers and AutoTokenizer in the Hugging Face stack, sentencepiece for the T5 and Llama 2 lineage, and llama-tokenize in llama.cpp.

WORDS47.7.6 remember these#

1. Token — a small piece of text the model works in — an integer id indexing one row of the embedding matrix, produced by the tokenizer.
2. Subword — a piece smaller than a word — a token that is part of a word, such as “ization” in “tokenization”.
3. Byte-pair encoding — repeatedly glue the most common pair together — BPE, a greedy merge algorithm producing an ordered merge list.
4. Byte-level — starting from raw bytes, not letters — using the 256 byte values as the base alphabet so no input is ever unrepresentable.
5. Pre-tokenization — the split done before merging — a regular expression pass that fixes where merges may not cross, such as across whitespace.
6. Vocabulary — the full list of possible tokens — the id-to-string table, size V, equal to the number of rows in the embedding matrix.
7. Special token — a marker that is not ordinary text — a reserved id such as sequence start or a chat role header, unreachable by merging user text.
8. Detokenization — turning tokens back into text — concatenating the token strings; exactly reversible in SentencePiece and byte-level BPE.

47.8 What tokenization does in practice#

PLAIN47.8.1 in simple words#

1. Now let us look at real text going through a real tokenizer.
2. All the counts below are measured, not estimated.
3. English prose splits neatly. Most common words are one token each.
4. Numbers get chopped into groups of digits, and the groups do not line up with how you would read the number.
5. Web addresses split into many small pieces, because slashes, dots and question marks all break the text apart.
6. Program code splits reasonably well, because tokenizers are trained on a lot of code these days.
7. Hindi and other Indian scripts split badly on older tokenizers and much better on newer ones, but still worse than English.
8. An emoji is one character to you and often two tokens to an older model.
9. Two consequences follow directly, and they explain famous model failures.
10. The model cannot see letters, so it cannot reliably count them or reverse a word.
11. The model sees digits in arbitrary groups, so arithmetic on long numbers goes wrong for reasons that have nothing to do with reasoning.

PLAIN47.8.2 a picture in your head#

1. Imagine reading a book through a narrow slot that shows a few characters at a time, but you do not control where the slot boundaries fall.
2. Somebody else cut the page into strips before handing it to you.
3. For English the strips mostly line up with words, so reading is easy.
4. For a number like 17394205 the strips fell at 173, 942, 05.
5. You can read the strips perfectly. You just never saw the digits as digits in their proper columns.
6. Now somebody asks you how many times the letter r appears in “strawberry”.
7. You never saw letters. You saw three strips: “str”, “aw”, “berry”.
8. You can guess, from having read the answer somewhere. You cannot count.

Where this comparison breaks: the model is not blind to letters in principle. It has seen enough spelled-out text during training to have learned a lot about which letters are inside which tokens. So it often gets these right. The point is that it is doing recall and inference rather than direct observation, which is why it fails unpredictably rather than never.

PLAIN47.8.3 a worked example#

1. Real measurements, taken with the actual OpenAI tokenizers. cl100k_base is the GPT-4 era encoding; o200k_base is the later one.

2. The exact splits, which are the interesting part:

"strawberry"  cl100k -> "str" "aw" "berry"
"strawberry"  o200k  -> "st" "raw" "berry"
"1234567890"  both   -> "123" "456" "789" "0"
"17394205"    both   -> "173" "942" "05"
"unbelievable" cl100k -> "un" "belie" "vable"
"tokenization" both  -> "token" "ization"

3. Look at “17394205”. The groups are 173, 942, 05. That is not thousands, hundreds, tens. It is an arbitrary cut.
4. Now the code example, tokenized by cl100k_base into 12 tokens:

"def" " add" "(a" "," " b" "):NL" "   " " return" " a" " +" " b"
(and a final newline token), where NL stands for a newline

5. Note that the four-space indent became a single token of three spaces plus a space attached to “return”. Indentation is compressed, which is why code tokenizes efficiently.
6. Now Devanagari. Take the Hindi sentence meaning “I live in India”, which is मैं भारत में रहता हूँ। In roman letters that is “main bharat mein rahta hoon”. It is 22 characters and 58 bytes of UTF-8, because each Devanagari character takes three bytes where an English letter takes one.

7. On the GPT-4 era tokenizer, the same meaning costs 24 tokens in Hindi and 5 in English. That is 4.8 times more, for identical content.
8. A longer measured pair: a 57-character Hindi sentence about tokenization costs 61 tokens on cl100k_base and 21 on o200k_base. The equivalent English sentence costs 11 on both.
9. So on cl100k_base the Hindi is 5.5 times the cost. On o200k_base it is 1.9 times. The gap narrowed a lot but did not close.
10. Emoji: the grinning face character, code point U+1F600, is one character and four UTF-8 bytes. cl100k_base spends 2 tokens on it. o200k_base spends 1.

PLAIN47.8.4 what is really happening inside#

1. Why does Devanagari cost so much on the older tokenizers?
2. Because the tokenizer’s merges were learned from a corpus that was mostly English. Frequent English sequences got merged into single tokens.
3. Devanagari sequences were rare, so few merges were learned for them.
4. What is left is the byte fallback. Each Devanagari character is three bytes in UTF-8, and if no merge covers it, those three bytes become separate tokens.
5. That is the 3x penalty in its rawest form, before you even count the missing word-level merges.
6. Newer tokenizers were trained on far more multilingual text and given much larger vocabularies, so many whole Hindi words now have their own token. That is why o200k_base needed only 6 tokens where cl100k_base needed 24.
7. Now the letter-counting failure, mechanically.
8. The model receives the id for “berry” as a single integer. It receives no information whatsoever about the five letters inside it, except what it learned indirectly during training.
9. Asking it to count the letter r in “strawberry” is asking it to recall a fact about the internal composition of three token ids.
10. It can often do this, because spelling appears in training data. It fails when the tokens are unusual, or when the counting has to be exact.
11. Reversing a word is worse, because it requires the letters in order, which is even less directly available.
12. Arithmetic fails for the same class of reason. When 17394205 arrives as 173, 942, 05, the place values that column-wise addition depends on are not aligned with anything the model can see.
13. This is why some model families deliberately force digits to be split one at a time, and why models are much better at arithmetic when allowed to write the working out step by step rather than answering in one shot.

TECHNICAL47.8.5 the engineer’s version#

1. Measured compression ratios, cl100k_base, on real text samples:

2. The common rules of thumb, roughly 4 characters per token and roughly 0.75 words per token in English, are conventions rather than standards, and the table shows they hold for prose and fail badly for structured data. JSON at 2.76 characters per token is 45 per cent worse than the rule of thumb suggests, which matters when you are budgeting a prompt full of API output.
3. The unfairness is documented. Aleksandar Petrov, Emanuele La Malfa, Philip Torr and Adel Bibi published Language Model Tokenizers Introduce Unfairness Between Languages at NeurIPS 2023, measuring the same content across many languages and finding token-count ratios up to 15 times between the best and worst served languages, with over 4 times difference remaining even for character and byte-level encoders.
4. The three consequences they name are exactly the ones that hurt an Indian developer: cost, because APIs bill per token; latency, because more tokens means more forward passes; and effective context, because a fixed window holds far less meaning in a penalized language.
5. Work the third one out. A 128,000-token window at 4.8 characters per token holds about 614,000 characters of English. At the cl100k_base Hindi rate of about 0.93 characters per token, the same window holds about 119,000 characters of Hindi. The window is the same. The content it holds is a fifth.
6. Mitigations that actually work: choose a model whose tokenizer was trained with substantial data in your language, measure with the real tokenizer rather than trusting a rule of thumb, and check whether the provider’s multilingual pricing is per token, which it almost always is.
7. On the letter-counting failure: the standard explanation is tokenization, and it is largely right, but be precise. Character-level models also make counting mistakes, and models can spell tokens correctly when asked step by step. So tokenization makes the information indirect rather than absent. That is an important distinction and it is often stated too strongly.
8. On arithmetic: the tokenizer’s digit grouping is a genuine and measurable cause. Research on numeric tokenization has shown that forcing single-digit tokenization, or right-to-left digit grouping, measurably improves multi-digit arithmetic. Some model families now tokenize digits individually for this reason.
9. Practical tools: tiktoken.get_encoding("cl100k_base").encode(text) returns the ids; enc.decode_single_token_bytes(i) shows exactly what each id is; Hugging Face’s tokenizer.tokenize(text) shows the string pieces.

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
ids = enc.encode("The cat sat on the mat.")
print(len(ids), ids)
for i in ids:
    print(i, enc.decode_single_token_bytes(i))

WORDS47.8.6 remember these#

1. Compression ratio — how much text fits in one token — measured characters per token or tokens per word for a given tokenizer and text type.
2. Byte fallback — what happens when no merge covers the text — encoding characters as their individual UTF-8 bytes, one token each.
3. Tokenizer fertility — how many tokens a tokenizer spends per word — a standard fairness measure; higher fertility means a worse-served language.
4. Digit grouping — how numbers get cut up — the pre-tokenizer rule that splits runs of digits, commonly into groups of up to three.
5. UTF-8 — the standard way text becomes bytes — 1 byte for ASCII, 2 for Latin accented and Greek, 3 for Devanagari and most CJK, 4 for emoji.
6. Effective context — how much meaning your window actually holds — the window size divided by your language’s tokens-per-meaning cost.

47.9 Token economics#

PLAIN47.9.1 in simple words#

1. When you use a model through a company’s service, you pay per token.
2. Not per question. Not per minute. Per token in and per token out.
3. This is not arbitrary. Tokens are what the machine actually does work on.
4. One token in costs the model one pass through the multiply-and-add machinery, though those can be done for many tokens at once.
5. One token out costs a full pass through the entire model, and those cannot be done in parallel, because each output token depends on the one before.
6. That asymmetry is why output tokens cost more than input tokens.
7. Typically output is priced at four to six times input.
8. So a long question with a short answer is cheap. A short question with a very long answer is not.
9. There is a second, sneakier cost. Long conversations resend everything.
10. If you have a chat with 50 turns, every new question sends all 50 previous turns as input, because the model has no memory between requests.
11. So the input token count grows with the conversation and you pay for the whole history every single time.
12. That is the single most common reason a bill is bigger than expected.

PLAIN47.9.2 a picture in your head#

1. Think of a translator you hire, who charges by the page.
2. Pages they read cost you a little. Pages they write cost you more, because writing is slower than reading.
3. That is input versus output pricing.
4. Now the important twist. This translator has no memory at all.
5. Every time you ask a follow-up question, you must hand them the entire previous conversation to read again, from the beginning.
6. On question one you hand over one page. On question fifty you hand over sixty pages, and pay to have all sixty read again.
7. Some translators offer a discount: if you hand them the exact same bundle of pages you handed them a few minutes ago, they charge much less for those.
8. That discount is prompt caching, and it is real.

Where this comparison breaks: a human translator reading sixty pages takes an hour. A model processes the input tokens of a long prompt in parallel, so sending a long history costs money and some latency but nothing like proportionally more wall-clock time than a short one.

PLAIN47.9.3 a worked example#

1. Build a realistic support assistant and cost it.
2. Every request contains: a system prompt of 800 tokens, four retrieved document chunks totalling 4,000 tokens, 1,200 tokens of conversation history, and the user’s 150-token question.
3. Input total: 800 + 4,000 + 1,200 + 150 = 6,150 tokens.
4. The answer averages 350 tokens of output.
5. Prices per million tokens, checked on 13 August 2026, input then output:

6. Work one out fully, for Claude Sonnet 5.
7. Input: 6,150 / 1,000,000 x $2.00 = $0.0123.
8. Output: 350 / 1,000,000 x $10.00 = $0.0035.
9. Total per request: $0.0158.
10. At 10,000 requests a day that is $158 a day, or $4,740 a month.
11. On Claude Opus 5 the same traffic is $11,850 a month.
12. On a small cheap model such as gpt-5.6-luna at $0.10 in and $0.60 out, the same traffic is $247.50 a month.
13. Notice that the input tokens dominate: $0.0123 of the $0.0158, or 78 per cent. Cutting retrieved context is worth far more than shortening answers.

PLAIN47.9.4 what is really happening inside#

1. Watch what happens when the context grows and everything else stays fixed.

2. Those are all the same question, the same answer, and the same model. The only change is how much context was stuffed in.
3. Going from 6,000 to 400,000 tokens of context multiplies the bill by 52.
4. This is why “just put the whole document in the context window” is a financial decision, not only a technical one.
5. Now caching. The system prompt and the retrieved chunks are often identical between requests within a short window.
6. Providers offer a large discount on input tokens that are an exact prefix match of a recent request. One current example prices cached input at one tenth of fresh input.
7. Rework the example on a provider charging $2.50 fresh, $0.25 cached and $15.00 output. Without caching: $0.0206 per request.
8. With the 4,800-token system-plus-documents prefix cached: 4,800 at $0.25 plus 1,350 at $2.50 plus 350 at $15.00 = $0.0098.
9. That is a 52 per cent saving for changing nothing but the ordering and stability of the prompt.
10. The rule that follows: put the stable parts of your prompt first and the variable parts last, because prefix caching only matches from the start.

TECHNICAL47.9.5 the engineer’s version#

1. Why output costs more, precisely. Processing input is the prefill phase: all input tokens go through the model in one batched forward pass, and the hardware is compute-bound and highly efficient.
2. Producing output is the decode phase: one token at a time, each requiring a full pass over every weight, and at small batch sizes the hardware is memory-bandwidth-bound and mostly idle in its arithmetic units.
3. So a decoded token costs far more machine time than a prefilled one, and the pricing reflects that. The typical ratio of 4x to 6x is a convention that has held across providers, not a standard.
4. Reasoning models add a third category. Tokens generated internally before the visible answer are billed as output tokens even though you never see them, and they can be several times the visible answer’s length. Budget for them explicitly.
5. Long-context surcharges are now common. As of August 2026, one provider prices prompts over 200,000 tokens at double the input rate and 1.5 times the output rate; another prices a long-context tier at double input and 1.5 times output above a threshold. Check the specific model’s page, because this changes often.
6. Batch APIs typically halve the price in exchange for asynchronous delivery within a window, commonly 24 hours. If your workload is not interactive, this is the largest single discount available.
7. Estimating before you build: tokenize a representative sample of your real prompts with the actual tokenizer and take the mean. Do not estimate from character counts if your content is JSON, code or a non-Latin script, because the rule of thumb is wrong by up to a factor of five in those cases.
8. A checklist that reduces token spend, in order of typical impact:

1. Retrieve less. Four chunks instead of twenty.
2. Cache the stable prefix; put variable text last.
3. Summarize old conversation turns instead of resending.
4. Use a small model for routing and a large one only when needed.
5. Cap max output tokens per request.
6. Batch anything that is not interactive.
7. Strip whitespace and boilerplate from injected documents.

9. Self-hosting changes the shape of the cost, not always its size. You pay for the machine by the hour whether it is busy or not, so self-hosting wins at high steady utilization and loses badly at low or spiky utilization.
10. Observability: every provider returns a usage object with input and output token counts per response. Log it per request with a feature label, or you will not be able to attribute the bill later.

WORDS47.9.6 remember these#

1. Input token — a token you send — a token in the prompt, processed in the prefill phase, billed at the lower rate.
2. Output token — a token the model writes — a token produced in the decode phase, billed at the higher rate, including hidden reasoning tokens.
3. Prefill — reading the prompt — the parallel forward pass over all input tokens, populating the key-value cache.
4. Decode — writing the answer — the sequential per-token generation loop, one full forward pass per token.
5. Prompt caching — paying less for a repeated prefix — reusing a stored key-value cache for an exact matching prompt prefix, at a discounted rate.
6. Batch API — cheaper if you can wait — asynchronous processing at roughly half price with a delivery window, commonly 24 hours.
7. Context stuffing — putting everything in the prompt — a strategy whose cost grows linearly in tokens and whose attention cost grows faster.

47.10 Embeddings: turning a token id into a vector#

PLAIN47.10.1 in simple words#

1. After tokenization you have a list of integers. The model cannot use those directly.
2. An integer id like 8415 is just a name. Token 8416 is not “one more than” token 8415 in any meaningful way.
3. So each id has to be turned into something with actual structure.
4. It is turned into a list of numbers. A long one: often 4,096 numbers.
5. That list is called the token’s embedding.
6. Where does the list come from? A big table.
7. The table has one row per possible token, and one column per number in the list.
8. If there are 128,256 possible tokens and each embedding is 4,096 numbers, the table is 128,256 rows by 4,096 columns.
9. Turning an id into an embedding is just reading row number 8415 of the table.
10. That is a lookup. No multiplication, no cleverness. Fetch the row.
11. The numbers in the table were learned during training, so the table is made of parameters, exactly like every other matrix in the model.
12. And that is the entrance to the model. Text became ids. Ids became rows of numbers. Now the layers can do arithmetic.

PLAIN47.10.2 a picture in your head#

1. Think of a very long index card box, with one card per word.
2. On each card, instead of a definition, there are 4,096 numbers.
3. To look up a word you find its card and copy the numbers off it.
4. The numbers mean nothing to you individually. Number 1,872 on the card for “cat” is 0.031, and there is no sentence explaining why.
5. But the cards are arranged in a very particular way in an imaginary space.
6. Cards for similar things ended up near each other. The card for “cat” is close to the card for “dog”, and far from the card for “invoice”.
7. Nobody sorted them deliberately. They ended up there because training pushed words used in similar ways toward similar numbers.
8. So the meaning is in the arrangement, not in any single number on any card.

Where this comparison breaks: index cards sit on a table, which is two dimensional. These cards sit in a space with 4,096 directions. Almost every intuition you have about “near” and “far” from two dimensions gets weaker in 4,096, and some of them break completely. That gets its own honest treatment below.

PLAIN47.10.3 a worked example#

1. Take a real sentence and follow it in.
2. The sentence: “The cat sat on the mat.”
3. Tokenized with cl100k_base it becomes 7 tokens with these real ids:

4. Note that “The” at the start and " the" in the middle are different tokens with different ids, because one has a leading space and a capital letter.
5. Now the lookup. The embedding matrix for an 8B-class model is 128,256 x 4,096.
6. Row 791 is fetched. It is 4,096 numbers. Row 8415 is fetched. Also 4,096.
7. Do that for all seven ids and stack them.
8. The result is a block of numbers 7 rows by 4,096 columns.
9. In the shape notation engineers use, the tensor went from [7] to [7, 4096].
10. That is 28,672 numbers describing a six-word sentence.
11. Size of the table itself: 128,256 x 4,096 = 525,336,576 parameters.
12. At two bytes each, that one table is about 1.05 gigabytes, which is 6.5 per cent of the whole 8B model.

PLAIN47.10.4 what is really happening inside#

1. The lookup is mathematically a matrix multiplication, done the lazy way.
2. The formal version: make a vector of 128,256 numbers that is all zeros except a single 1 in position 8415. That is called a one-hot vector.
3. Multiply that one-hot vector by the embedding matrix. The result is row 8415.
4. Every implementation skips the multiplication and just indexes the row, because multiplying by 128,255 zeros is a waste.
5. That is why the operation is called an embedding lookup rather than an embedding multiply, even though it is a linear layer.
6. One thing is missing at this point: order.
7. The lookup gives the same vector for “cat” whether it is the first word or the fiftieth. Position information has to be added separately.
8. Older models added a learned position vector to the token vector. Modern models mostly rotate the query and key vectors inside attention instead, which Chapter 48 covers.
9. From here the stacked embeddings flow into layer 1, then layer 2, and so on.
10. Critically, the vector at each position keeps being updated as it passes through layers. It starts as “the meaning of this token in general” and becomes “the meaning of this token in this sentence”.
11. At the very end, the vector at the last position is multiplied by the output matrix to produce one score per possible token, and the highest scores are the model’s prediction of what comes next.

TECHNICAL47.10.5 the engineer’s version#

1. The embedding matrix has shape [V, d] and is exactly V x d parameters.

2. In PyTorch this is nn.Embedding(V, d), whose forward pass is F.embedding, which is an index-select, not a matmul.
3. The output projection, lm_head, has shape [V, d] as well and produces logits: one raw score per vocabulary entry. Softmax over logits gives the next-token distribution.
4. If embeddings are tied, lm_head.weight is the same storage as embed_tokens.weight. Tying was popularized by Press and Wolf in 2016 in Using the Output Embedding to Improve Language Models, and independently by Inan, Khosravi and Socher the same year.
5. Common hidden sizes, which are also the embedding dimensions: 768 for GPT-2 small, 1024 and 1600 for larger GPT-2 variants, 2048 for many 1B-class models, 4096 for 7B and 8B, 5120 for 13B, 8192 for 70B, 16384 for 405B.
6. Now the honest note about dimensionality that the reader asked for.
7. You cannot visualize 4,096 dimensions. Nobody can. Any picture you have seen of word vectors as arrows on a page is a projection down to two dimensions by a method such as PCA, t-SNE or UMAP, and those projections distort distances substantially. t-SNE in particular preserves local neighbourhoods and does not preserve global distances at all, so cluster sizes and the gaps between clusters in a t-SNE plot are not meaningful.
8. What does carry over from 2D and 3D: dot products, cosine of the angle between vectors, Euclidean distance, addition and subtraction of vectors, and the idea that direction can encode something.
9. What does not carry over, and this is the part people get wrong:
   1. In high dimensions almost every pair of random vectors is nearly orthogonal, so a cosine of 0 means “unrelated”, not “opposite”.
   2. Distances concentrate: the ratio between the nearest and farthest point in a random set shrinks toward 1 as dimension grows, which is one face of the curse of dimensionality.
   3. The volume of a high-dimensional ball is almost entirely near its surface, so intuitions about “the middle of the cloud” are wrong.
   4. There is far more room than you expect. In d dimensions you can pack exponentially many nearly-orthogonal directions, which is why a 4,096-wide space can hold far more than 4,096 distinguishable concepts.
10. That last point has a name in current research: superposition, the hypothesis that models represent many more features than they have dimensions, by using nearly-orthogonal directions and tolerating small interference. Anthropic’s 2022 paper Toy Models of Superposition is the standard reference. This is active research, not settled fact.

WORDS47.10.6 remember these#

1. Embedding — a token turned into a list of numbers — a row of the embedding matrix, a vector in R^d.
2. Embedding matrix — the lookup table — a [V, d] parameter tensor mapping token ids to vectors.
3. One-hot vector — all zeros with a single 1 — the formal input to an embedding layer, never materialized in practice.
4. Logits — raw scores before they become probabilities — the [V] output of the final projection, converted by softmax.
5. Dimension — how many numbers are in one vector — d, the model’s hidden size, typically 768 to 16,384.
6. Curse of dimensionality — high-dimensional space behaves strangely — the family of effects including distance concentration and near-orthogonality of random vectors.
7. Superposition — packing more ideas than there are directions — the hypothesis that features are stored along nearly-orthogonal directions with tolerated interference. Active research.

47.11 What embeddings capture#

PLAIN47.11.1 in simple words#

1. If two words have similar embeddings, the model treats them similarly.
2. So the natural question is: how do you measure “similar” for two lists of numbers?
3. Two standard answers, both simple.
4. Distance: treat each list as a point, and measure how far apart the points are. Closer means more similar.
5. Angle: treat each list as an arrow from the origin, and measure the angle between the arrows. Smaller angle means more similar.
6. The angle version is the one almost everybody uses, and it is called cosine similarity.
7. It gives 1 for arrows pointing the same way, 0 for arrows at right angles, and -1 for arrows pointing opposite ways.
8. The angle version ignores how long the arrows are and looks only at direction, which is usually what you want.
9. There is also a famous claim that these vectors capture relationships, not just similarity.
10. The claim is that king minus man plus woman lands near queen.
11. It is a real effect. It is also considerably weaker than the popular story, and the popular demonstration was rigged in a specific way. Section 47.11.3 and 47.11.5 give the honest account.

PLAIN47.11.2 a picture in your head#

1. Think of a map of a country, where each town is a dot.
2. Two towns close together on the map are similar in location. That is distance.
3. Now draw an arrow from the capital to every town.
4. Two towns roughly in the same direction from the capital, one near and one far, have arrows at a small angle. That is cosine similarity.
5. On a map, “50 km north-east” is a direction you can apply anywhere. Start at any town, go 50 km north-east, arrive somewhere new.
6. The analogy claim is that word space has directions like that. There is a direction that means “make it female”, and applying it to king lands on queen.
7. The map picture makes clear why this could work and why it might not.
8. On a real map, “50 km north-east” from a coastal town lands in the sea.

Where this comparison breaks: a map has two dimensions and every direction is meaningful. Word space has thousands, and only a tiny number of directions correspond to anything a human would name. Most directions in the space mean nothing at all.

PLAIN47.11.3 a worked example#

1. Cosine similarity, computed by hand on small vectors.
2. Take four-number toy vectors, made up for illustration:

cat = [0.8, 0.1, 0.7, 0.2]
dog = [0.7, 0.2, 0.8, 0.1]
car = [0.1, 0.9, 0.0, 0.8]

3. The formula: cosine = (a . b) / (|a| x |b|), where a . b is the sum of pairwise products and |a| is the square root of the sum of squares.
4. Step 1, the dot product of cat and dog: 0.8x0.7 + 0.1x0.2 + 0.7x0.8 + 0.2x0.1 = 0.56 + 0.02 + 0.56 + 0.02 = 1.16.
5. Step 2, the length of cat: sqrt(0.64 + 0.01 + 0.49 + 0.04) = sqrt(1.18) = 1.0863.
6. Step 3, the length of dog: also 1.0863.
7. Step 4: cosine = 1.16 / (1.0863 x 1.0863) = 1.16 / 1.18 = 0.9831.
8. An angle of 10.6 degrees. Very similar.
9. Now cat against car. Dot product = 0.08 + 0.09 + 0.00 + 0.16 = 0.33. Lengths 1.0863 and 1.2083. Cosine = 0.33 / 1.3126 = 0.2514.
10. An angle of 75.4 degrees. Nearly unrelated.
11. Compare with straight-line distance: cat to dog is 0.20, cat to car is 1.41. Both measures agree here, which is common but not guaranteed.
12. Now the analogy, on a three-number toy set:

king     = [0.92, 0.81, 0.12]
man      = [0.88, 0.14, 0.09]
woman    = [0.13, 0.16, 0.91]
queen    = [0.15, 0.74, 0.86]
princess = [0.20, 0.60, 0.88]

13. king - man + woman = [0.17, 0.83, 0.94].
14. Cosine of that result against every word in the toy set:

15. Queen wins. The demonstration works. But look at the third row: woman, one of the inputs, scores 0.8556, well above king and man.
16. That third row is where the honest correction lives.

PLAIN47.11.4 what is really happening inside#

1. The standard way to answer an analogy from vectors is called 3CosAdd.
2. Compute b - a + c, then find the vocabulary word whose vector has the highest cosine with the result.
3. There is a rule buried in the standard implementation: the three input words are excluded from the search.
4. So when you compute king - man + woman, the code refuses to answer king, man or woman, no matter how well they score.
5. That matters a great deal, because very often the highest-scoring word is one of the inputs.
6. The result you are shown is therefore the best answer after the most likely answers have been forbidden.
7. That does not make the effect fake. Queen really does score highly, and consistently, on many word pairs.
8. It makes the effect smaller and less magical than the demonstrations imply.
9. When researchers removed the exclusion rule and let the inputs compete, accuracy collapsed. The numbers are in the technical block below.
10. There is a second, separate honesty point about these vectors.
11. In word2vec, each word has exactly one vector, forever. “Bank” gets one vector that has to serve both the river bank and the money bank.
12. Modern models do not work that way. The vector for a word changes depending on the sentence it appears in.

TECHNICAL47.11.5 the engineer’s version#

1. Word2vec was published by Tomas Mikolov and colleagues at Google in 2013, in two papers: Efficient Estimation of Word Representations in Vector Space in January 2013, and Distributed Representations of Words and Phrases and their Compositionality at NeurIPS 2013. The analogy result appeared in a third, Linguistic Regularities in Continuous Space Word Representations, by Mikolov, Yih and Zweig at NAACL in June 2013.
2. GloVe followed from Jeffrey Pennington, Richard Socher and Christopher Manning at Stanford, EMNLP 2014, fitting vectors to global co-occurrence counts rather than local context windows.
3. The vector offset method, 3CosAdd, ranks candidate d by cos(d, b - a + c). Omer Levy and Yoav Goldberg proposed 3CosMul in 2014 in Linguistic Regularities in Sparse and Explicit Word Representations, which multiplies similarities instead of adding offsets and performs better.
4. Now the correction, which is well documented and not fringe.
5. Tal Linzen’s 2016 RepEval paper Issues in Evaluating Semantic Spaces Using Word Analogies showed that a large part of reported analogy accuracy is explained by simple baselines: for many analogy items, just returning the nearest neighbour of c, ignoring a and b entirely, scores well.
6. Malvina Nissim, Rik van Noord and Rob van der Goot published Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor in Computational Linguistics, volume 46 issue 2, 2020. They showed the standard implementations forbid returning any of the three input words, and that this is not a detail.
7. Their measured result: when they modified the algorithms to allow input words as answers, 3CosAdd accuracy fell from 71 per cent to 21 per cent, and 3CosMul fell from 73 per cent to 45 per cent.
8. Read that carefully. On the standard benchmark, roughly two thirds of 3CosAdd’s apparent success disappears when the exclusion is lifted. The frequent correct-if-allowed answer is one of the inputs, usually b.
9. So the accurate statement is: the offset direction carries real relational information, and it is much weaker than the “king - man + woman = queen” story suggests. Both halves of that sentence are true, and popular accounts usually give only the first half.
10. The same paper makes a separate methodological point about bias claims: asking a system “man is to doctor as woman is to what” while forbidding it from answering “doctor” guarantees a different answer, and then reporting that different answer as evidence of bias is unsound. Bias in embeddings is real and separately demonstrated; that particular demonstration is not sound evidence of it.
11. Static versus contextual embeddings. Word2vec and GloVe are static: one vector per word type, fixed. Contextual embeddings give a different vector per occurrence.
12. ELMo, from Matthew Peters and colleagues at the Allen Institute, NAACL 2018, Deep Contextualized Word Representations, was the first widely used contextual model. BERT, from Jacob Devlin and colleagues at Google, posted October 2018 and published at NAACL 2019, made it standard.
13. In a transformer, the row fetched from the embedding matrix is static, and it becomes contextual immediately: after the first attention layer the vector at that position has mixed in information from the other tokens.
14. So in the sentence “I sat on the river bank” and “I went to the bank for a loan”, the input embedding for “bank” is byte-for-byte identical, and the representation at layer 12 is quite different. That is the whole point of the architecture.

WORDS47.11.6 remember these#

1. Cosine similarity — how aligned two lists of numbers are — the dot product divided by the product of the norms; 1 identical direction, 0 orthogonal.
2. Euclidean distance — straight-line distance between two points — the square root of the sum of squared differences.
3. Dot product — pairwise multiply and add — the unnormalized similarity, which grows with vector length as well as alignment.
4. Vector offset — subtracting one vector from another — the b - a term in the 3CosAdd analogy method, treated as a relation direction.
5. Static embedding — one fixed vector per word — word2vec or GloVe style, a pure lookup with no sentence context.
6. Contextual embedding — a different vector each time — the hidden state at a position after attention has mixed in surrounding tokens.
7. Polysemy — one word with several meanings — the phenomenon static embeddings cannot represent and contextual embeddings can.

47.12 Embeddings as a tool in their own right#

PLAIN47.12.1 in simple words#

1. So far embeddings have been a part inside a language model.
2. They are also a useful product on their own, sold separately.
3. The idea: instead of one vector per token, get one vector for a whole sentence, paragraph or document.
4. Now you can compare pieces of text by comparing numbers.
5. That single capability unlocks five very common jobs.
6. Semantic search: find documents that mean the same thing as a query, even if they share no words with it.
7. Clustering: group thousands of support tickets into themes automatically.
8. Recommendation: suggest articles near the ones a reader liked.
9. Deduplication: find near-identical items that keyword matching misses.
10. Retrieval for a model: fetch the few relevant paragraphs from a large collection and paste them into the prompt.
11. That last one is the foundation of retrieval-augmented generation, which Chapter 51 covers in full.
12. The thing that makes all five work is the same: meaning became geometry, and geometry is cheap to compute with.

PLAIN47.12.2 a picture in your head#

1. Imagine a library where every book has been assigned a position in a very large warehouse.
2. Books are not shelved by author or title. They are shelved by what they are about.
3. Two books about monsoon farming end up on adjacent shelves even if one is in English and one in Hindi and they share no words.
4. To answer a question you do not read anything. You compute where the question would sit in the warehouse, walk to that spot, and take the nearest ten books.
5. That is semantic search, exactly.
6. Keyword search is the old catalogue: it finds books whose title contains your exact words, and misses everything phrased differently.

Where this comparison breaks: a warehouse has aisles you can walk. A vector space has thousands of dimensions and no aisles, so finding the nearest items needs a specialized index. Doing it by brute force means comparing against every item, which is fine for ten thousand documents and far too slow for a hundred million.

PLAIN47.12.3 a worked example#

1. Suppose four short documents and one query, each already turned into a four-number vector by an embedding model.

query = [0.75, 0.15, 0.60, 0.20]

doc A "how to feed a kitten"  = [0.80, 0.10, 0.70, 0.20]
doc B "puppy training basics" = [0.70, 0.20, 0.80, 0.10]
doc C "car insurance renewal" = [0.10, 0.90, 0.00, 0.80]
doc D "truck loading limits"  = [0.20, 0.80, 0.10, 0.90]

2. Compute the cosine of the query against each.
3. The query’s length is sqrt(0.5625 + 0.0225 + 0.36 + 0.04) = sqrt(0.985) = 0.9925.
4. Query and doc A: dot = 0.60 + 0.015 + 0.42 + 0.04 = 1.075. Doc A’s length is 1.0863. Cosine = 1.075 / (0.9925 x 1.0863) = 0.9971.
5. Query and doc B: dot = 0.525 + 0.03 + 0.48 + 0.02 = 1.055. Cosine = 0.9786.
6. Query and doc C: dot = 0.075 + 0.135 + 0.00 + 0.16 = 0.37. Cosine = 0.3085.
7. Query and doc D: dot = 0.15 + 0.12 + 0.06 + 0.18 = 0.51. Cosine = 0.4196.

8. The two animal documents rank above the two vehicle documents, and no word from the query appears in any of them.
9. In a real system the vectors have 384, 768, 1024 or 3072 numbers instead of four, and there are millions of documents, but the computation per pair is exactly this one.

PLAIN47.12.4 what is really happening inside#

1. Where does a whole-sentence vector come from?
2. Run the text through a model and you get one vector per token. You need one vector for the whole thing.
3. Three common ways to collapse them.
4. Mean pooling: average all the token vectors. Simple, works well, still the most common method.
5. CLS pooling: use the vector at a special marker token placed at the start, which was trained to summarize the sequence.
6. Last-token pooling: in a decoder-only model, use the vector at the final position, which has seen everything before it.
7. But a raw language model’s vectors are not well suited to comparison straight out of the box.
8. So embedding models are trained specially, with a method called contrastive training.
9. You show the model many pairs. Some pairs mean the same thing, some do not.
10. You push the matching pairs’ vectors together and the non-matching pairs’ vectors apart.
11. After enough pairs, cosine similarity in that space genuinely tracks meaning similarity, which it does not automatically do otherwise.
12. Finally, vectors are usually normalized to length 1, which makes cosine similarity and dot product the same computation, and makes the search faster.

TECHNICAL47.12.5 the engineer’s version#

1. Embedding models are separate, smaller models from the generative ones, and they are usually encoder-style rather than decoder-style.
2. Typical output dimensions and what they cost to store, per million documents:

3. Sentence-BERT, by Nils Reimers and Iryna Gurevych, EMNLP 2019, is the paper that made sentence embeddings practical, by fine-tuning BERT with a siamese network so that cosine similarity between sentence vectors is meaningful. Before it, comparing raw BERT sentence vectors performed worse than averaging GloVe vectors.
4. Contrastive objectives in common use: InfoNCE, multiple-negatives ranking loss, and in-batch negatives, where every other item in the batch is treated as a negative example for free.
5. Matryoshka representation learning, from a 2022 paper by Aditya Kusupati and colleagues, trains a model so that the first 256 or 512 numbers of a 3,072- number vector are usable on their own. Several current embedding APIs expose this as a dimension parameter, letting you trade accuracy for storage without re-embedding.
6. Nearest-neighbour search at scale uses an approximate index rather than brute force. HNSW, Hierarchical Navigable Small World graphs, from Yury Malkov and Dmitry Yashunin in 2016, is the most widely deployed. IVF with product quantization, from the FAISS library released by Facebook AI Research in 2017, is the other common family.
7. Approximate means approximate: these indexes return the true nearest neighbours most of the time, not always, and the recall is a tunable parameter traded against speed. That is a real correctness consideration and is often glossed over.
8. Vector databases and vector indexes in wide use as of 2026 include FAISS, pgvector for PostgreSQL, Qdrant, Weaviate, Milvus, Chroma and the vector features of Elasticsearch and OpenSearch.
9. Retrieval-augmented generation, named in a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, is embedding search plus prompt construction: embed the query, retrieve the top k chunks by cosine, paste them into the context, and let the generative model answer from them. Chapter 51 covers the whole pipeline including chunking and reranking.
10. Honest limitation: pure vector search is worse than keyword search at exact matching, such as product codes, error numbers and names. Production systems almost always combine both, which is called hybrid search, usually merged with reciprocal rank fusion.
11. A second honest limitation: an embedding has no notion of truth or recency. Two documents that contradict each other on a fact will sit close together, because they are about the same thing.

WORDS47.12.6 remember these#

1. Embedding model — a model whose output is a vector, not text — usually an encoder trained with a contrastive objective for similarity.
2. Pooling — turning many token vectors into one — mean, CLS or last-token aggregation over the sequence.
3. Contrastive training — pull matches together, push non-matches apart — an objective such as InfoNCE using in-batch negatives.
4. Semantic search — search by meaning, not words — ranking documents by cosine similarity between query and document embeddings.
5. Approximate nearest neighbour — fast, nearly-correct nearest search — index structures such as HNSW or IVF-PQ that trade recall for speed.
6. Hybrid search — combining keyword and vector search — merging BM25 and vector rankings, commonly with reciprocal rank fusion.
7. Normalization — scaling every vector to length 1 — makes dot product equal cosine similarity and speeds up search.

47.13 The context window#

PLAIN47.13.1 in simple words#

1. The context window is the maximum number of tokens the model can be given at once, including everything: system instructions, history and your question.
2. It is measured in tokens, not words and not characters.
3. It is a hard limit set by the model, not a preference.
4. Why is there a limit at all? Because of how attention works.
5. Attention lets every token look at every other token.
6. With 100 tokens that is 10,000 pairs. With 1,000 tokens, a million pairs.
7. Double the length and the number of pairs goes up four times, not two.
8. So cost grows with the square of the length. Chapter 48 works through why.
9. When you exceed the window, one of two things happens.
10. Either the service refuses the request with an error, or something silently gets cut off, usually the oldest part of the conversation.
11. Silent cutting is the dangerous one, because the model then answers confidently based on text it never received.
12. And there is a further catch. A model that accepts a long input does not necessarily use all of it well.

PLAIN47.13.2 a picture in your head#

1. Think of a desk with a fixed area.
2. You can spread out papers on it, and you can read anything on the desk instantly.
3. But when the desk is full and you add another page, one falls off the edge.
4. Nobody tells you which one fell off, or that anything fell at all.
5. A bigger desk helps. It also costs more, and it takes longer to scan.
6. Now the awkward part. Even with a huge desk, if you spread out four hundred pages, you will reliably notice the first few and the last few and be much vaguer about the middle.
7. That is not a memory limit. It is an attention limit, and models have a measurable version of the same thing.

Where this comparison breaks: a desk keeps papers between work sessions. A context window does not persist at all. When the request ends, everything on the desk is swept away, and the next request starts empty. Anything that appears to persist was resent by the application.

PLAIN47.13.3 a worked example#

1. Work out what a window actually holds.
2. At about 4 characters per token, a 128,000-token window holds roughly 512,000 characters of English.
3. A typical printed page is about 2,000 characters, so that is about 256 pages.
4. But you never get all of it. Reserve space for the answer, and for the system prompt.
5. Now the attention arithmetic, which is why longer costs so much more:

6. Going from 512 to 1,048,576 tokens is 2,048 times the length and about four million times the attention pair count.
7. Memory grows too, but linearly, through the key-value cache: 40 GiB for a 70B model at 128,000 tokens at 16-bit, as computed in section 47.4.
8. So a million-token context is not a small engineering step from a thousand-token one. It requires different algorithms, not just more memory.

PLAIN47.13.4 what is really happening inside#

1. What happens when the input is too long? Four common strategies.
2. Reject: return an error. Honest, and the only one that cannot silently mislead.
3. Truncate: cut the input to fit. Usually the oldest turns go first. Cheap, and information is genuinely lost.
4. Sliding window: keep a moving window of the most recent tokens and drop what falls out of the back. Used in streaming settings.
5. Summarize and compact: replace old turns with a shorter summary generated by the model, keeping recent turns verbatim. This is what most long-running assistants do.
6. Each strategy loses something, and you should know which one your framework is using, because they fail differently.
7. Now the harder problem. Suppose the whole document does fit.
8. The model still does not attend evenly across it.
9. Researchers put a single fact somewhere in a long context and asked a question that requires it, varying only the position.
10. Accuracy was highest when the fact was at the very beginning or the very end, and lowest when it was in the middle.
11. The shape of the curve is a U. The finding is called “lost in the middle”.
12. So placing your most important instruction in the middle of a long prompt is a measurably bad idea.

TECHNICAL47.13.5 the engineer’s version#

1. Context windows have grown by more than three orders of magnitude in seven years:

2. As of 13 August 2026, current commercial figures: Claude Opus 5, Claude Sonnet 5 and Claude Fable 5 accept 1,000,000 input tokens with up to 128,000 output tokens; Claude Haiku 4.5 accepts 200,000. The GPT-5.6 family accepts about 1.05 million input tokens, which is 2^20, with 128,000 output. These figures change often; check the provider’s model page.
3. The cost of attention is O(n^2) in time for the score matrix and O(n) in memory for the key-value cache. FlashAttention, from Tri Dao and colleagues in 2022, does not change the O(n^2) compute but avoids materializing the n x n matrix in high-bandwidth memory, which removes the O(n^2) memory term and is what made long contexts practical. Chapter 48 covers the mechanism.
4. Position encoding is the second limit. RoPE, rotary position embedding from Jianlin Su and colleagues in 2021, encodes position by rotating query and key vectors. Extending a trained model beyond its training length needs the rotation frequencies adjusted, by methods such as position interpolation or the NTK-aware and YaRN scalings. A model advertised at 128,000 tokens is often a model trained at 8,000 and extended.
5. Lost in the middle: Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang, arXiv July 2023, published in TACL 2024. Across multi-document question answering and key-value retrieval, performance was highest when the relevant information was at the start or end of the input and degraded significantly in the middle, and extended-context models were not reliably better than their shorter counterparts at using the extra length.
6. RULER, from Cheng-Ping Hsieh and colleagues at NVIDIA, April 2024, extended the needle-in-a-haystack test with multi-hop tracing and aggregation tasks and evaluated 17 long-context models. Its headline finding: although all claimed 32,000 tokens or more, only about half maintained satisfactory performance at 32,000, despite near-perfect scores on the simple retrieval test.
7. So state the distinction plainly. Claimed context is the number the API will accept without an error. Effective context is the length at which the model still performs near its short-context accuracy. The second number is smaller, is model-specific, and is rarely published by vendors.
8. The needle-in-a-haystack test, popularized by Greg Kamradt in November 2023, is a weak measure on its own: passing it shows the model can find one exact verbatim string, which is much easier than reasoning over dispersed information.
9. Practical rules that follow: put critical instructions at the beginning and repeat them at the end; retrieve fewer, better chunks rather than more; measure your own task at your own lengths rather than trusting the advertised number; and log token counts so you notice when truncation starts.
10. Tools: every API returns input token counts in its usage object; Anthropic and OpenAI both expose a token counting endpoint or library so you can check before sending; tiktoken and AutoTokenizer let you count offline.

WORDS47.13.6 remember these#

1. Context window — the most tokens the model accepts at once — the maximum input sequence length, a fixed architectural and training limit.
2. Truncation — cutting input to make it fit — dropping tokens, usually oldest first, often silently.
3. Sliding window — a moving view of recent tokens — keeping the last n tokens and discarding what falls out.
4. Lost in the middle — the dip in accuracy for content in the centre — the U-shaped position-versus-accuracy curve documented by Liu and colleagues.
5. Effective context — how much of the window is actually usable — the length at which task accuracy still holds up, usually shorter than the claimed window.
6. RoPE — how position is encoded in modern models — rotary position embedding, rotating query and key vectors by an angle proportional to position.
7. Needle in a haystack — the simple long-context test — planting one verbatim fact and asking for it back; necessary but far from sufficient.

47.14 Putting it together: one prompt from text to numbers#

PLAIN47.14.1 in simple words#

1. Everything in this chapter meets in one short journey.
2. You type a sentence. It is characters.
3. The tokenizer chops it into pieces and looks each piece up, giving integers.
4. Each integer picks one row out of the embedding table, giving a list of numbers.
5. Those lists are stacked into a block, one row per token.
6. The block goes into layer 1, comes out changed, goes into layer 2, and so on.
7. Each layer transforms the block using its own frozen parameters.
8. At the end, the row belonging to the last token is turned into one score for every possible next token.
9. The scores are turned into probabilities, one is picked, and that becomes the next token.
10. Then the whole thing repeats with that token added on the end.
11. Nothing in the model changed at any point. Only the numbers flowing through.

PLAIN47.14.2 a picture in your head#

1. Picture a factory line with 32 stations.
2. At the entrance, a machine cuts the incoming text into standard-sized parts and stamps each with a part number.
3. A stores clerk takes each part number and fetches the matching tray of 4,096 components.
4. The trays travel down the line. Every station reworks all the trays a little, using its own fixed jigs and tools.
5. At the last station, the final tray is compared against a catalogue of 128,256 possible products, and the closest matches are scored.
6. One is chosen and shipped. Then the whole line runs again with that product added to the input.

Where this comparison breaks: on a factory line each station works on one item at a time. In a transformer every station works on all positions at once, and crucially each position can look at the others, which no factory station does.

PLAIN47.14.3 a worked example#

1. The prompt: “The cat sat on the mat.” Model: an 8B-class model with hidden size 4,096, 32 layers, vocabulary 128,256.

stage 0  raw text
         "The cat sat on the mat."     23 characters

stage 1  tokenize (cl100k_base)
         7 tokens
         791, 8415, 7731, 389, 279, 5634, 13
         "The" " cat" " sat" " on" " the" " mat" "."
         shape: [7]

stage 2  embedding lookup
         fetch rows 791, 8415, ... from a
         [128256, 4096] table
         shape: [7, 4096]   = 28,672 numbers

stage 3  through layer 1 ... layer 32
         shape stays [7, 4096] at every layer boundary
         inside a layer it widens to [7, 14336] in the
         feed-forward middle and comes back

stage 4  final normalization
         shape: [7, 4096]

stage 5  take the last row only, position 6
         shape: [4096]

stage 6  multiply by the output matrix [128256, 4096]
         shape: [128256]      one logit per token

stage 7  softmax -> probabilities, then sample
         one token id comes out, say 13 for "."

stage 8  append and repeat from stage 2 with 8 tokens

2. Real sizes at each stage, at 16-bit: stage 2 holds 28,672 numbers, which is 56 KiB. Stage 6 produces 128,256 numbers, which is 251 KiB.
3. The key-value cache after this prompt holds 7 tokens x 128 KiB = 896 KiB.
4. Every one of those numbers is an activation. Every number in the tables they passed through is a parameter.

PLAIN47.14.4 what is really happening inside#

1. Notice the two places where the vocabulary size appears.
2. Once at the entrance, as a table you index into. Once at the exit, as a table you multiply against.
3. Those two are the same shape and are sometimes literally the same numbers.
4. Notice that the shape [7, 4096] is stable through the whole stack.
5. That stable shape is called the residual stream. Every layer reads from it and writes back into it by adding.
6. Notice that only the last row is used to predict the next token, during generation. The other rows were computed anyway, and their keys and values are cached so they do not need recomputing next time.
7. Notice what the second pass costs. It processes one new token, not eight. All previous work is in the cache.
8. That is why the first token of a reply is slower than the rest: the first one pays for the whole prompt, and each later one pays for a single token.

TECHNICAL47.14.5 the engineer’s version#

1. The shapes with a batch dimension, which is how it really runs:

2. Logits are usually computed in fp32 even in a bf16 model, because softmax over 128,256 entries is numerically sensitive.
3. During prefill, S is the full prompt length and all positions are computed. During decode, S is 1 and only the new position is computed, with the cache supplying the rest.
4. Per-token generation cost in floating point operations is approximately 2 x N, where N is the parameter count, ignoring attention over the cache. For an 8B model that is about 16 billion operations per token.
5. Prompt processing cost is approximately 2 x N x S, so a 6,150-token prompt on an 8B model is about 98 trillion operations, which is why prefill is compute-bound and decode is bandwidth-bound.
6. Memory traffic per decoded token at batch 1 is roughly the entire weight file, because every weight is read once. At 4-bit that is about 4.5 GiB per token for an 8B model, so a machine with 100 GB/s of memory bandwidth caps out near 21 tokens per second regardless of how fast its arithmetic is.
7. That single fact explains most local inference performance questions, and it is why quantization speeds up generation as well as shrinking it.
8. Inspecting the journey in practice: tokenizer(text, return_tensors="pt") gives stage 1; model.get_input_embeddings()(ids) gives stage 2; model(ids, output_hidden_states=True).hidden_states gives every stage 3 boundary as a tuple of 33 tensors for a 32-layer model.

WORDS47.14.6 remember these#

1. Residual stream — the stable block of numbers every layer reads and writes — the [B, S, d] tensor threaded through the network by residual connections.
2. Prefill — processing the prompt — one parallel forward pass over all input positions, filling the key-value cache.
3. Decode — producing one token — a forward pass with sequence length 1 reusing the cache.
4. Logit — a raw score for one possible next token — one entry of the [V] output vector before softmax.
5. Softmax — turning scores into probabilities — exponentiate and divide by the sum, giving a distribution over the vocabulary.
6. Time to first token — how long before the reply starts — the latency of prefill, growing with prompt length.
7. Memory bandwidth bound — limited by moving numbers, not by computing — the regime single-stream decoding runs in, where tokens per second is set by bytes read per token.

47.98 Common wrong ideas#

1. Wrong: more parameters means a smarter model. Right: parameter count sets a ceiling and predicts memory and speed very well, but Chinchilla at 70B beat Megatron-Turing NLG at 530B, and Mistral 7B beat Llama 2 13B. Data quantity, data quality, training compute and post-training all matter as much or more.
2. Wrong: a token is a word. Right: a token is a subword piece. English averages about 1.1 to 1.3 tokens per word, JSON averages over 3, and a Hindi sentence can cost 24 tokens where the English equivalent costs 5.
3. Wrong: the context window is the model’s memory, and a model that accepts 1,000,000 tokens uses all of them equally. Right: it is the input size limit for one request, nothing persists between requests, any apparent memory is your application resending the history, and accuracy dips for content in the middle, with RULER finding only about half of models claiming 32,000 tokens performed well at that length.
4. Wrong: quantization makes a model dumber in proportion to the bits removed. Right: the curve is very flat then steep. Going 16-bit to 8-bit cost 0.01 perplexity on a measured Llama-3.1-8B test; 16-bit to Q4_K_M cost 0.24; going to Q3_K_S cost 1.64. The first two thirds of the saving is nearly free.
5. Wrong: embeddings understand meaning. Right: they encode statistical co-occurrence as position in a space. That is enough for similarity to be useful and is not understanding. Two contradictory documents about the same fact sit close together, because they are about the same thing.
6. Wrong: hyperparameters are just small parameters. Right: hyperparameters are chosen by a human before training and are never learned. Parameters are learned by gradient descent and are never chosen.
7. Wrong: a 7B model needs 14 GB to train. Right: 14 GB is inference at 16-bit. Full training with Adam needs about 18 bytes per parameter for states alone, which is 117 GiB for 7B, before activations.
8. Wrong: king minus man plus woman equals queen proves embeddings capture relationships cleanly. Right: it is a real but weak effect, and the standard evaluation forbids returning the input words. With that exclusion removed, 3CosAdd accuracy fell from 71 per cent to 21 per cent.
9. Wrong: the model cannot count letters because it is stupid. Right: it receives “strawberry” as three integer ids and never sees individual letters. It answers from indirect knowledge, which is why it succeeds sometimes and fails unpredictably.
10. Wrong: input and output tokens cost the same. Right: output is typically priced four to six times higher, because generating is sequential and bandwidth-bound while reading the prompt is parallel and compute-bound.

47.99 Chapter summary in 20 lines#

1. A parameter is one learned number sitting in one position of one matrix, set during training and frozen thereafter; “weight” and “parameter” are used interchangeably, with biases being additive parameters rather than weights.
2. A model’s parameter count is the sum of the sizes of all its matrices, and it is computable by hand from five published shape numbers.
3. Worked exactly, Llama 3.1 8B holds 8,030,261,248 parameters: 32 layers of 218,112,000 plus two 525,336,576 vocabulary tables plus a final norm.
4. The billions live in the feed-forward blocks: 70 per cent for 8B, 80 per cent for 70B, 81 per cent for 405B, with attention near 17 per cent throughout.
5. Hyperparameters are pre-training human choices, activations are temporary numbers created while running, context is the input text, tokens are the pieces it is chopped into. Only parameters are in the model file.
6. Memory equals parameter count times bytes per parameter: 7B is 26.1 GiB at fp32, 13.0 at fp16, 6.5 at int8 and 3.3 at int4.
7. Inference needs more than the weights: a key-value cache of 320 KiB per token for a 70B model, plus activations and framework overhead.
8. Training needs about 18 bytes per parameter for weights, master copy, gradients and Adam’s two moments, so a 7B model needs about 117 GiB of state before activations are counted.
9. Quantization stores each weight in fewer bits using a shared scale and optional zero point, per tensor, per channel or per group of 32 to 256.
10. GPTQ, AWQ, NF4 and the GGUF k-quants are the common formats; Q4_K is exactly 4.5 bits per weight, and Q4_K_M averages about 4.8 because sensitive tensors are kept at Q6_K.
11. A 70B model at Q4_K_M needs about 39.7 GiB of weights and fits a 64 GiB machine at a modest context; at fp16 it needs 131.4 GiB and does not.
12. Capability does not scale linearly with parameters: Chinchilla in 2022 showed models were undertrained, recommended about 20 tokens per parameter, and a 70B model beat a 530B one at equal compute.
13. Text is tokenized into subwords because whole words give an unbounded vocabulary and characters give sequences that are far too long.
14. Byte-pair encoding repeatedly merges the most frequent adjacent pair; on a four-word corpus, ten merges produce a 21-token vocabulary in which “newest” is a single token and “est_” emerges as a suffix without anyone naming it.
15. WordPiece merges by likelihood gain, Unigram prunes down from a large candidate set, and SentencePiece is the framework that makes both work without language-specific pre-tokenization.
16. English runs about 4 characters or 1.1 to 1.3 tokens per word; JSON is far worse; and a Hindi sentence costing 5 tokens in English cost 24 on the GPT-4-era tokenizer and 6 on the later one, which is real, measurable cost and context inequality.
17. Models cannot reliably count letters, reverse words or do long arithmetic because they never see letters or aligned digits, only token ids.
18. An embedding turns a token id into a row of a [V, d] table; the numbers mean nothing individually and everything collectively, and 4,096 dimensions cannot be visualized, with distance concentration and near-orthogonality breaking most 2D intuitions.
19. Similarity is measured by cosine of the angle; the king minus man plus woman result is real but much weaker than told, since removing the input-word exclusion dropped 3CosAdd accuracy from 71 to 21 per cent.
20. The context window is limited because attention cost grows with the square of length; exceeding it truncates or errors; and accepting a long context is not the same as using it well, as the lost-in-the-middle and RULER results show.