What is a token in AI? The unit your whole bill is written in
- glossary
Third glossary entry, and the most mercenary one. Models do not read words, sentences or characters. They read tokens, and everything practical about working with an LLM, what fits, what it costs, where it gets slow, is measured in them. If AI were a utility, the token would be the kilowatt hour, and like the kilowatt hour, nobody thinks about it until the bill arrives.
- The plain definition. A token is the chunk a model splits text into before processing: usually a short word, part of a longer word, or a piece of punctuation.
- The rule of thumb. In English, a token is about three-quarters of a word; 1,000 tokens is roughly 750 words. Code and other languages cut less evenly.
- Why it exists. Models compute over a fixed vocabulary of chunks, tens or hundreds of thousands of them, not over an open-ended dictionary of words.
- Where you feel it. Context windows are token limits, pricing is per token, in and out separately, and latency grows with token count.
- The habit worth building. Estimate before you paste. A 43 KB file is not "a file", it is roughly ten thousand tokens of budget, every single time.
The plain definition
Before a model sees your text, a tokenizer slices it into pieces from a fixed vocabulary learned alongside the model. Common words survive as single tokens. Longer or rarer words get split: "monitoring" might be one token, "observability" two or three. Spaces and punctuation count. The model then reads and generates token by token, which is also why streaming output arrives in little bursts rather than whole sentences.
The exact slicing differs between model families, so the same paragraph can be 980 tokens for one vendor and 1,040 for another. The differences are small enough that the working rule holds across vendors: English prose runs about four characters or three-quarters of a word per token. Other languages and code cut less evenly, usually into more tokens for the same amount of meaning.
What it means in practice
Everything you will ever negotiate with an LLM project runs on token meters.
The context window is a token limit: the maximum the model can have in front of it at once, prompt, pasted material, conversation history and its own answer combined. What happens at the limit depends on where you hit it. A raw API call that overflows fails with a hard error. A chat product or an agent framework avoids that error by quietly trimming, usually the oldest material, and whatever it trimmed, the model never saw. That trimming is why long conversations start forgetting their own beginnings.
The bill is per token, input and output priced separately, output usually several times dearer. This is why verbose prompts and chatty answers are not a style question. At automation scale, where an agent runs a prompt hundreds of times a day, a few thousand unnecessary tokens per run compound into real money.
Latency follows token count too, with an asymmetry worth knowing: input is absorbed roughly in one go, while output is generated piece by piece, so a long answer slows you far more than a long question. The fastest optimization in most agent pipelines is not a better model, it is asking for less back.
What people get wrong
| People say | What is true |
|---|---|
| A token is a word | Close enough for napkin math, wrong at the edges: long words split, punctuation counts, code tokenizes worse |
| Only the answer costs money | Input tokens are billed too. A giant pasted document with a one-line question is mostly input cost |
| The window is in characters or pages | It is in tokens. A "500-page" window means nothing until you tokenize the pages |
| Repeating context is free | Every repetition rides the window and the invoice. Caching helps with cost and speed, not with the window |
| Token limits are a vendor trick | The limit is mostly architectural: working over the whole window is what makes the model work, and it gets expensive fast, which is also why longer windows cost more per token |
One example from my own work
The number that made tokens concrete for me came out of auditing my own agent config. The main instruction file had grown to 43 KB of markdown, which felt like "a text file", something a computer should not even notice. Tokenized, it is roughly ten thousand tokens, loaded into the window at the start of every single session, before the first word of the actual task. A two-minute question paid the same ten-thousand-token toll as a two-hour build. That is the shape of the cost that matters: not one expensive load, but a fixed toll on every session, including all the sessions that needed none of it. The audit exists because of that arithmetic, and the arithmetic only became visible when I stopped thinking in kilobytes and started thinking in tokens.
Where to go deeper
Entry one drew the map: What is AI. Entry two covered the engine that reads these tokens: What is an LLM. The working end of token economics, what to load always versus on demand, is the whole subject of the CLAUDE.md audit. Next in the glossary: what happens when the token predictor fills a gap with something that was never true.