Key takeaways
- A token is a piece of text, often part of a word, that a language model reads and writes instead of raw letters.
- Google's Gemini docs give about 4 characters, or 60 to 80 English words per 100 tokens, as a rough estimate; exact counts come from the provider's counting tool.
- Tokens set API cost and fill the context window, and Anthropic says its newer tokenizer yields about 30 percent more tokens for the same text.
Tokens in AI are the small pieces of text a language model reads and writes. A token can be a whole word, part of a word, or a punctuation mark, and the model never sees your text any other way. Everything an AI service measures, limits and charges for is counted in tokens, so it helps to know what they are.
What a token is
A language model can't work with a raw string of letters. Before your prompt reaches the model, a tokenizer splits it into pieces and turns each piece into a number. The model reads those numbers and produces numbers back, which are turned into text again.
The pieces are usually not whole words. Common words tend to be one piece, and rare words get cut into several. The idea goes back to a 2015 paper, Neural machine translation of rare words with subword units, by Rico Sennrich, Barry Haddow and Alexandra Birch. Translation models had a fixed vocabulary and no good answer for words outside it. The authors proposed writing rare and unknown words as sequences of smaller subword units, which lets a model handle words it has never seen. They compared several ways of cutting words, including byte pair encoding, and report gains of 1.1 BLEU for English to German and 1.3 BLEU for English to Russian over a dictionary fallback on the WMT 15 test sets.
We read the paper's abstract page, not the full text. What matters here is the principle: a fixed list of pieces, small enough to manage, that can still spell out any word.
How many tokens is my text
Providers give rough rules, and they differ by model. Google's Gemini docs say that for Gemini models, a token is equivalent to about 4 characters, and that 100 tokens is about 60 to 80 English words. Treat that as an estimate. Code, numbers and languages other than English often split differently.
For an exact count, ask the provider. Anthropic's token counting endpoint takes the same input as a normal message, including system prompts, tools, images and PDFs, and returns the number of input tokens. It is free to use, with its own rate limit (5,000 requests per minute on the Start tier, as of November 8, 2026). Anthropic calls the result an estimate that can differ by a small amount from the real count, and says you are not billed for tokens it adds for system optimizations. Gemini has a countTokens call for the same job, and its responses report input, output, thinking, cached and tool-use tokens in a usage field.
The same text can cost a different number of tokens on a different model. Anthropic's docs say Claude 4.7 and later models use a newer tokenizer that produces about 30 percent more tokens for the same text, depending on content, and tell developers to recount prompts against the model they plan to use.
Tokens are not only text
Images, audio and video turn into tokens too. According to Google's Gemini docs:
| Input | Tokens |
|---|---|
| Image up to 384 pixels on both sides | 258 |
| Larger image | 258 per 768x768 tile |
| Video | 263 per second (about 6,000 for 60 seconds at default resolution) |
| Audio | 32 per second (about 1,920 for 60 seconds) |
Anthropic's counting endpoint likewise accepts images and PDFs. For its own example it shows 1,028 input tokens for one image plus the words "Describe this image". Other providers count differently.
Why tokens drive cost
API pricing is per token. Google's docs say the cost of a Gemini API call is determined in part by the number of input and output tokens, and that system instructions and tool definitions count as input. Prices are usually quoted per million tokens, with output costing more than input. Our AI API pricing page lists current rates for the main providers.
Two practical consequences follow. A long system prompt is paid for on every request. And a chatty answer costs more than a terse one. Counting tokens before you send a request, as both Anthropic and Google suggest, is the cheapest way to avoid a surprise bill.
Models that think before answering add another bucket. Anthropic's docs note that thinking from the current assistant turn counts toward input tokens, and Gemini reports thinking tokens separately in its usage data.
Why tokens set the limit on context
A model can only read so many tokens at once. That limit is its context window, and your prompt, the conversation so far, any attached files and the model's own answer all have to fit inside it. We explain it in what is a context window. Because words become tokens at a ratio you can't see, the honest way to know whether a document fits is to count it.
Tokens also explain why repeated text is such a target for savings. If the first 50,000 tokens of every request are the same, you can pay less to resend them. That is what prompt caching does. Anthropic's counting endpoint does not apply caching logic, so its numbers are estimates without it.
What to do with this
- Count with the provider's own tool, not a rule of thumb, before you plan a budget.
- Recount when you change models, because tokenizers differ.
- Remember that images, audio, video, tool definitions and system prompts all use tokens.
- Cut tokens you don't need: trim the prompt, cap the answer length, reuse stable prefixes.
Prices and limits change often. Check the provider's pricing page before you rely on a number here.
Frequently asked questions
How many words is a token?
Google's Gemini docs say 100 tokens is about 60 to 80 English words and a token is about 4 characters. It is an estimate and varies by model and language.
Is counting tokens free?
On Anthropic's API, token counting is free, with its own requests-per-minute limits by usage tier. Billing happens on the actual message request.
Do images use tokens?
Yes. Gemini's docs say an image up to 384 pixels per side costs 258 tokens, and larger images are tiled at 258 tokens per 768x768 tile.
Why do different models count different tokens for the same text?
Each model family has its own tokenizer. Anthropic says Claude 4.7 and later models produce about 30 percent more tokens for the same input than earlier ones.
