Key takeaways
- AI inference is running a trained model on new input to produce an output; training is how the model learned its weights in the first place.
- A reply is generated in two phases: a parallel prefill that reads the prompt, then a memory-bound decode that writes one token at a time using a KV cache.
- As of November 16, 2026, Anthropic charges $2 per million input tokens and $10 per million output tokens for Claude Sonnet 5.5, with a 50% discount for batch jobs.
AI inference is what happens when a trained model is put to work: it takes a new input, such as your prompt, and computes an output, such as the reply. NVIDIA defines it as "the process where a trained AI model generates new outputs by reasoning and making predictions on new data."
Training happens once per model. Inference happens every time anyone uses it, so its speed and cost decide what an AI product can charge.
AI inference vs training
Training is how a model learns. Data passes through the network, its guesses are compared with the right answers, and its weights, the billions of numbers that make up the model, are adjusted over many rounds of error feedback. NVIDIA describes it as very compute-intensive.
Inference uses those weights as they are. Nothing is learned. The model reads your input, produces an output, and its weights stay the same.
What happens for each token
A language model writes one token, a chunk of text often shorter than a word, at a time. Answering a prompt takes two phases.
Prefill. The model reads the whole prompt in one pass, processing all input tokens in parallel. According to NVIDIA's guide to inference optimization, this keeps the GPU fully busy.
Decode. The model then writes the reply one token at a time, because each token depends on the ones before it. NVIDIA calls decode "a memory-bound operation": for every token, the chip reads the model's weights from memory, and moving that data takes longer than the math.
The KV cache
In a transformer, each token produces a key and a value vector per layer, which later tokens use to decide what to pay attention to. Hugging Face's docs explain the waste this creates: to predict the 1,000th token the model needs the previous 999, and for the 1,001st it needs them all again.
The KV cache stores those keys and values so each decode step only computes them for the newest token. Without it, the attention work per step grows with the square of the sequence length.
The cost is memory. NVIDIA's example: Llama 2 7B at 16-bit precision with a batch of one needs about 2 GB of KV cache, next to roughly 14 GB of weights. The cache grows with conversation length and with the number of users served at once, and the Hugging Face cache guide warns it can become the bottleneck for long contexts.
Why inference is priced per million tokens
The work scales with the tokens a model reads and writes, so providers bill by the token, quoted per million because one token costs a fraction of a cent. Here is Anthropic's API pricing as of November 16, 2026:
| Model | Input, per million tokens | Output, per million tokens | Cached input reads |
|---|---|---|---|
| Claude Haiku 5.5 (prompts up to 100,000 tokens) | $0.10 | $0.50 | $0.01 |
| Claude Sonnet 5.5 | $2 | $10 | $0.10 |
| Claude Opus 5.5 | $4 | $20 | $0.20 |
The table lines up with the mechanics above.
- Output costs five times as much as input. The page gives no reason, but the mechanics suggest one: input is read in one parallel prefill, while each output token needs its own memory-bound decode step.
- Cached input is far cheaper. A repeated prompt prefix lets the provider reuse work instead of running prefill again. For Sonnet 5.5, a cache hit costs 0.05 times the base input price.
- Batch jobs are half price. Anthropic's Batch API takes 50% off input and output for requests that can wait, which are easier to pack into large batches.
The AI API pricing page compares every major provider.
How providers make inference faster and cheaper
Batching
Since decode spends most of its time reading weights, serving one user at a time wastes the chip. Batching runs many requests together, so each read of the weights produces tokens for all of them. NVIDIA recommends in-flight, or continuous, batching, which swaps finished requests for new ones as it goes.
Batch size is usually capped by KV cache memory. The 2023 vLLM paper by Woosuk Kwon and colleagues tackled that with PagedAttention, which stores the cache in fixed-size blocks, borrowing paging from operating systems. The authors report near-zero wasted cache memory and 2 to 4 times the throughput of systems such as FasterTransformer and Orca at the same latency.
Speculative decoding
Speculative decoding, from Yaniv Leviathan, Matan Kalman and Yossi Matias (ICML 2023), attacks the one-token-at-a-time limit. A small draft model guesses several tokens ahead, and the big model checks all of them in one parallel pass, keeping the guesses up to the first wrong one. The output matches what the big model would have written alone. The paper reports a 2 to 3 times speedup on T5-XXL "with identical outputs."
The idea is still moving. MMFlash, posted to arXiv on November 13, 2026 by Sicheng Feng, Xinchao Wang and colleagues, applies it to vision-language models. They found draft guesses are accepted 46% of the time on tokens that depend least on the image, but 22% on those that depend most. MMFlash trains its drafter to focus on those tokens and drafts 16 tokens per pass. The authors report lossless speedups of up to 3.58 times, mainly on Qwen3.5-4B and Qwen3.5-9B on one NVIDIA H200. These are their own results, not independent tests.
Quantization
Quantization stores a model's numbers at lower precision, such as 8 or 4 bits instead of 16. NVIDIA notes most models work with about eight bits per value. Since decode is limited by memory traffic, fewer bytes per weight means faster tokens and fewer chips per model.
The hardware: GPUs, TPUs and inference chips
GPUs. Graphics processors, most of them NVIDIA's, are built for the large matrix multiplications that both training and inference rely on. NVIDIA ships serving software, TensorRT-LLM and NIM, to run on them.
TPUs. Google's Tensor Processing Units are, per Google Cloud's docs, "custom-developed, application-specific integrated circuits." Their core is a 128 by 128 systolic array, a grid of multipliers that pass results to their neighbors, with high-bandwidth memory (HBM) on the chip.
Inference chips. Some chips only run models. Amazon's Inferentia2 offers up to 190 TFLOPS of 16-bit compute and 32 GB of HBM per chip. Groq's LPU keeps data in on-chip SRAM (not to be confused with Grok); Groq lists a rack of 256 LPUs with 128 GB of SRAM, sold as NVIDIA Groq 3 LPX.
Because decode is memory-bound, how much fast memory a chip has matters as much for inference as raw compute.
This week: an inference cloud that tunes itself
Chips set the ceiling; software decides how close a provider gets. Isoquant, a Y Combinator company, launched on November 16, 2026 with what it calls "a self-optimizing inference cloud." It says automated agents tune GPU kernels, memory, decoding, caching and scheduling for each customer's workload.
Isoquant claims it serves open models 4 times faster than other providers, at over 1,200 tokens per second with latency under half a second. These are the company's own numbers, with no independent tests and no public prices yet. The claims target the same levers covered above, batching, caching and decoding, which is where the next cut in the price per million tokens is likely to come from.
Frequently asked questions
What is the difference between AI inference and training?
Training adjusts a model's weights over many rounds of error feedback and happens before the model ships. Inference uses the finished weights, unchanged, to produce outputs for new inputs, every time someone sends a prompt.
Why are AI inference costs priced per million tokens?
The work a model does grows with the tokens it reads and writes, so providers bill by the token, quoted per million because one token costs a fraction of a cent. Output tokens usually cost more than input, for example $10 versus $2 per million for Claude Sonnet 5.5 as of November 16, 2026.
What chips are used for AI inference?
Mostly GPUs, especially NVIDIA's, plus Google's TPUs and chips built only for inference, such as Amazon's Inferentia2 and Groq's LPU. Because generating tokens is limited by memory speed, high-bandwidth memory matters as much as raw compute.
What is speculative decoding?
A small draft model guesses several tokens ahead and the large model checks them all in one parallel pass, keeping the ones it agrees with. The original 2023 paper reported 2 to 3 times faster generation on T5-XXL with identical outputs.
Sources
- What's the Difference Between Deep Learning Training and Inference?NVIDIA
- Mastering LLM Techniques: Inference OptimizationNVIDIA
- How caching worksHugging Face
- Cache strategiesHugging Face
- Efficient Memory Management for Large Language Model Serving with PagedAttentionarXiv (Kwon et al.)
- Fast Inference from Transformers via Speculative DecodingarXiv (Leviathan et al.)
- MMFlash: Faster Speculative Decoding for Vision-Language ModelsarXiv (Feng et al.)
- PricingAnthropic
- Introduction to Cloud TPUGoogle Cloud
- AWS InferentiaAmazon Web Services
- LPU architectureGroq
- Isoquant: self-optimizing inference cloud for 4x faster open modelsY Combinator



