Key takeaways

  • A token is the piece of text a model reads and writes, about 4 characters or 0.6 to 0.8 English words on average, according to Google's Gemini docs.
  • Most tokenizers use byte pair encoding, which Sennrich, Haddow and Birch brought to language models in a 2015 paper to handle rare and unseen words.
  • Counts differ by model: Anthropic says Claude Opus 4.7 and later produce about 30 percent more tokens for the same text than earlier Claude models.

The short answer to what is a token in AI: it is the small piece of text a language model reads and writes, a whole word like "model", part of a word like "ization", a single character, or a punctuation mark. In English a token averages about four characters, so 100 tokens come to roughly 60 to 80 words, according to Google's Gemini docs on tokens. API prices and context windows are both counted in tokens.

The exact count depends on the model's tokenizer, the language and the kind of text.

What a token is

Before your prompt reaches the model, a program called a tokenizer cuts it into pieces and swaps each piece for a number. The model sees only the numbers. Its answer comes back as numbers too, and the tokenizer turns them back into text.

Running OpenAI's open-source tiktoken library on the sentence "Tokenization splits text into pieces that a language model can read." gives 13 tokens:

Token | ization | ␣splits | ␣text | ␣into | ␣pieces | ␣that | ␣a | ␣language | ␣model | ␣can | ␣read | .

Common words get one token each, usually with the space in front of them included. The less common word "Tokenization" is split into two familiar parts. The rare word "unbelievably" becomes three: "un", "bel" and "ievably".

How tokenization works: byte pair encoding

The method behind many tokenizers, including OpenAI's tiktoken, is byte pair encoding, or BPE. It came to language AI through a 2015 paper by Rico Sennrich, Barry Haddow and Alexandra Birch, "Neural Machine Translation of Rare Words with Subword Units", which was accepted at ACL 2016.

Translation models then worked from a fixed list of whole words, so any name, compound or new word outside the list was unknown. The authors note that "translation is an open-vocabulary problem". Their fix was BPE, a data compression trick from 1994 that repeatedly replaces the most common pair of bytes with a new symbol. Applied to text, it works like this:

  1. Start with every word split into single characters, plus a marker for the end of the word.
  2. Count every pair of neighboring symbols across the training text.
  3. Merge the most frequent pair into one new symbol, such as "e" + "r" into "er".
  4. Repeat. Each merge adds one symbol to the vocabulary.

After thousands of merges, common words become single symbols and rare words are left as a few pieces. The number of merges is the method's only setting; the paper's vocabularies had 60,000 to 100,000 units. A word the tokenizer never saw, like "lower" in the paper's toy example, still gets split into known parts ("low" and "er"). Only characters never seen in training can stay unknown.

On the WMT 15 translation tasks, the subword models beat a dictionary-based baseline by 1.1 BLEU points (a standard translation score) for English to German and 1.3 for English to Russian.

The same idea runs today's tokenizers. OpenAI's tiktoken README says BPE is lossless, works on any text, even text the tokenizer never saw, and compresses text to about 4 bytes per token on average.

How many words is a token

The rule of thumb from Google's docs is about 4 characters per token, or 60 to 80 English words per 100 tokens. So one English word is about 1.3 to 1.7 tokens.

To test it, three OpenAI tokenizers from tiktoken 0.14.0 were run in a Linux VM on The Zen of Python, a 144-word English text that ships with Python. The "used by" column comes from tiktoken's own model mapping.

Tokenizer Used by Vocabulary Tokens Words per token
o200k_base GPT-4o 200,019 176 0.82
cl100k_base GPT-4 100,277 181 0.80
gpt2 GPT-2 50,257 206 0.70

The newer tokenizers have bigger vocabularies and fit the same text into fewer tokens.

Two kinds of text break the rule of thumb:

  • Other languages. The Slovenian translation of the example sentence above, "Tokenizacija razdeli besedilo na koščke, ki jih jezikovni model lahko prebere.", came to 24 tokens with o200k_base and 28 with cl100k_base, against 13 for the English. Both tokenizers cut the Slovenian words into smaller pieces than the English ones.
  • Numbers. o200k_base splits "1234567" into "123", "456" and "7".

Images, audio and video are counted in tokens too. Google's docs say an image of 384 pixels or less on both sides costs 258 tokens on Gemini, larger images are cut into 768-by-768 tiles of 258 tokens each, audio costs 32 tokens per second, and video 263 tokens per second.

Why prices are counted in tokens

Tokens are the unit the model works in, so they are the unit it is billed in. Every request has input tokens, everything you send, and output tokens, everything the model writes back. Output costs more.

As of November 15, 2026, Anthropic's pricing page lists Claude Opus 5.5 at $4 per million input tokens and $20 per million output tokens, and Claude Sonnet 5.5 at $2 and $10. Anthropic's own worked example: a session that uses 50,000 input tokens and 15,000 output tokens on Opus 5.5 costs $0.20 for the input and $0.30 for the output.

Anthropic's docs add that thinking tokens, the reasoning a model writes before its answer, are billed as output.

Prices across companies are side by side on the AI API pricing page, and how to check ChatGPT token usage shows where your own counts appear.

Why context windows are counted in tokens

A model's context window is how much it can hold in view at once, and it is measured in tokens. Anthropic's context window docs say everything in a request counts toward it: the system prompt, every message, tool results, images, documents and tool definitions, plus the answer the model writes, including its thinking. Claude Opus 5.5 and Sonnet 5.5 have a 1-million-token window and can write up to 128,000 output tokens in one request.

Using Google's rule of 60 to 80 words per 100 tokens, a million tokens is very roughly 600,000 to 800,000 English words. The context window explainer covers what happens when a conversation fills it.

How token counts differ between models

Each model family has its own tokenizer, so a count from one model can't be reused on another. Anthropic's token counting docs say Claude Opus 4.7 and later models use a newer tokenizer that produces "approximately 30 percent more tokens" for the same text than earlier Claude models, with the exact increase depending on the content. Billing follows the new count, and Anthropic tells developers to recount prompts on the new model.

This matters when comparing prices. A lower price per token isn't a cheaper request if the tokenizer splits your text into more pieces, so compare token counts on your own text as well as the price.

How to count tokens

  • Claude: the token counting endpoint takes the same input as a normal message and returns the input token count. It is free, and Anthropic calls the result an estimate.
  • Gemini: the API's count_tokens returns the tokens in the input only.
  • OpenAI models: tiktoken counts tokens locally, with tiktoken.encoding_for_model("gpt-4o") picking the right tokenizer.

The counts in this article came from pip install tiktoken and short Python scripts calling get_encoding() and encode() on the samples above.

For a fuller picture of what happens to tokens once they are inside the model, see what is an LLM.

Frequently asked questions

How many words is 1,000 tokens?

Google's rule of thumb of 60 to 80 English words per 100 tokens puts 1,000 tokens at roughly 600 to 800 English words. Other languages, numbers and code usually fit fewer words into the same number of tokens.

What is tokenization in AI?

Tokenization is the step that cuts text into tokens and swaps each one for a number from the model's fixed vocabulary. Most modern tokenizers use byte pair encoding, which builds the vocabulary by repeatedly merging the most frequent pair of symbols in its training text.

Why do AI models charge per token?

Tokens are the unit the model reads and writes, so API prices are set per million input and output tokens. As of November 15, 2026, Claude Opus 5.5 costs $4 per million input tokens and $20 per million output tokens.

Do all AI models count tokens the same way?

No. Each model family has its own tokenizer, so the same text gives different counts. Count your text with the tokenizer or counting endpoint of the exact model you plan to use.

Sources

  1. Neural Machine Translation of Rare Words with Subword Units (Sennrich, Haddow, Birch)arXiv
  2. Understand and count tokensGoogle
  3. Token countingAnthropic
  4. Context windowsAnthropic
  5. PricingAnthropic
  6. tiktokenOpenAI

Add ClaudeGPT Si as a preferred source on Google