A token is roughly three-quarters of a word, the unit models actually process.
DefinitionWhat it means
A token is the basic unit of text a language model processes, produced by a tokenizer that splits words into subword pieces, roughly three to four characters or about three-quarters of a word on average in English. Rare or made-up words may split into several tokens, while common words often form a single token, and tokenization schemes differ across model families, which is why the same text can cost different amounts across providers.
Why it mattersWhy you should care
Tokens are the unit of everything commercial about LLMs: API pricing, context-window limits, and latency are all measured per token, so understanding tokenization directly affects cost estimation and prompt design. Product teams also need to know that non-English languages, code, and unusual formatting often tokenize less efficiently, which can quietly inflate costs and eat into the effective context budget.
At a glanceSee it
How a byte-pair tokenizer learns its vocabulary — it loops, merging the most frequent adjacent pair until the target size is reached, and rare words still shatter into many pieces.
Why the three-quarters-of-a-word rule is only an average — real token counts swing with word frequency, spacing, digits, and writing system.
Where you see itIn the wild
- API pricing pages billed per thousand tokens
- A tokenizer playground showing how a sentence gets split
- Context-window limits stated as a token count