What is a token?

A token is the unit of text a language model actually reads. It is not a word and not a character — it sits somewhere in between. Common words are usually a single token. Rarer words get split into pieces. Punctuation and spaces count too.

Tokens matter for two very practical reasons: context windows are measured in them, and API bills are calculated from them.

Why models don't just use words

A vocabulary of whole words has an awkward problem: it can never be complete. New words, names, typos and technical jargon would all be unrepresentable.

The standard fix is subword tokenization — an approach introduced for machine translation and now used almost everywhere. Frequent words get their own token; anything unfamiliar is assembled from smaller fragments. The model can therefore represent any string at all, while keeping common text compact.

The part that costs you money

Tokenizers are trained mostly on English text, so English is the most efficient thing to feed them. The same meaning, expressed in a different language or in code, can cost noticeably more tokens.

Input	Why the token count moves
Ordinary English prose	Most efficient. Common words are one token each.
Technical jargon and rare names	Split into fragments — several tokens per word.
Code	Whitespace, indentation and symbols all consume tokens.
Languages not written in Latin script	Often several tokens per character.
Long numbers	Frequently split digit-by-digit.

The last row surprises people the most: a table of figures can be far more expensive than a paragraph of the same visual size.

Rules of thumb

English prose runs roughly three-quarters of a word per token — a loose convention, not a constant. It moves with the tokenizer: Anthropic's newer models emit about 30% more tokens for the same text, dragging that ratio closer to half a word per token.
You pay for input and output tokens, usually at different rates. Output is typically the more expensive of the two.
On the API, the whole conversation is re-sent every turn, so input tokens accumulate much faster than people expect.

What to actually do

If you're building something, don't estimate — count. Every major vendor ships a tokenizer library that gives exact numbers for their models, and the exact number is the one you'll be billed for. If you're just using a chat product, the practical version is simpler: pasting a large document into a conversation costs you that document on every subsequent message, not just the first.

What we addedA worked breakdown of why the same sentence costs different amounts in English, code and other languages, and what that means for your bill.

This article was researched and drafted with AI assistance from the sources listed below, then checked and edited by Fiqhro Dedhen before publication. How we work.