Key takeaways
- An LLM is a neural network trained on huge amounts of text to predict the next token; GPT-3 had 175 billion parameters and trained on 300 billion tokens.
- In the InstructGPT paper, raters preferred a 1.3-billion-parameter model tuned with human feedback over the 175-billion-parameter GPT-3.
- Context windows grew from 2,048 tokens in GPT-3 to about 1 million in current Claude, GPT-6 and Gemini 4 models, but hallucination remains.
An LLM, or large language model, is a program trained on a huge amount of text to do one thing: predict the next token, a word or a piece of a word, from the text before it. Claude, ChatGPT and Gemini run on LLMs tuned to follow instructions.
That one skill, practised hundreds of billions of times, is enough to translate, summarize and write code. Four papers from 2017 to 2022 lay out the recipe and explain where these models fail. (The LL.M. law degree is a different LLM.)
What is an LLM made of
An LLM is a neural network: a very large set of numbers, called parameters, that turns the text it is given into a probability for every possible next token. "Large" means many parameters and a lot of training text. GPT-3, described by OpenAI researchers in the May 2020 paper Language Models are Few-Shot Learners, had 175 billion parameters, 10 times more than any earlier model of its kind, its authors said.
Almost every LLM is built on the transformer, which eight Google researchers introduced in June 2017 in Attention Is All You Need. It relies on attention, a way for every word in the input to look at every other word at once, instead of reading one word at a time. Built for translation, it beat the best earlier English-to-German results by more than 2 BLEU points and trained in days on eight GPUs, because the work runs in parallel. Our transformer explainer covers how attention works, and our piece on mixture-of-experts models covers a variant that runs only part of the network for each token.
How an LLM is trained: next-token prediction at scale
The first stage is pretraining. The model reads a large pile of text and, at every position, guesses the next token. When it guesses wrong, its parameters are nudged so the right token becomes a little more likely. Nobody labels the data, because the text itself is the answer key.
GPT-3 was trained on 300 billion tokens, 60% of them from filtered pages of Common Crawl, a public web archive, and the rest from web text, books and English Wikipedia.
Why so big? In January 2020, Jared Kaplan and colleagues at OpenAI published Scaling Laws for Neural Language Models. They found that the model's prediction error falls along a smooth power law as model size, data and computing budget grow, with some trends holding across more than seven orders of magnitude. Larger models also learned more from each token, which gave labs a predictable reason to build bigger.
The GPT-3 paper showed what scale bought. Given a task description and a few examples in the prompt, with no extra training, the model handled translation, question answering and fill-in-the-blank tests. The authors call this few-shot learning.
Fine-tuning and RLHF turn a predictor into an assistant
A pretrained model continues text. It was never trained to follow instructions. The March 2022 InstructGPT paper from OpenAI, Training language models to follow instructions with human feedback, states the problem: "Making language models bigger does not inherently make them better at following a user's intent."
The fix has three steps:
- Supervised fine-tuning. People write good answers to prompts, and the model trains on them.
- A reward model. People rank the model's answers, and a second model learns to predict their preference.
- Reinforcement learning. The model is trained to produce answers the reward model scores highly, using an algorithm called PPO.
Steps 2 and 3 are reinforcement learning from human feedback, or RLHF. Raters preferred answers from a 1.3-billion-parameter InstructGPT model to those of the 175-billion-parameter GPT-3, which has the same architecture and more than 100 times the parameters. At equal size, they preferred InstructGPT 85% of the time, give or take 3 points. More in our fine-tuning explainer.
What is an LLM token, and what is a context window
An LLM never sees letters or whole words. It sees tokens, chunks of text cut by a tokenizer. Google's Gemini documentation puts a token at about 4 characters, with 100 tokens equal to roughly 60 to 80 English words. Tokenizers differ: Anthropic says 1 million tokens hold about 555,000 words on its current Claude tokenizer, against 750,000 on its earlier one.
The context window is how much the model can work with at once. Anthropic's documentation defines it as all the text a model can reference while generating a response, including the response, and calls it the model's "working memory." GPT-3's window was 2,048 tokens. The current flagship models from Anthropic, OpenAI and Google each take about a million. Anthropic also warns that accuracy and recall degrade as the token count grows. See our context window explainer.
What a model learned in pretraining stops at its training cutoff, which Anthropic lists as June 2026 for its current Claude models. Anything newer has to be put into the context window.
Examples of LLMs in November 2026
Current models as their makers describe them in their docs (Anthropic, OpenAI, Google, Gemma), as of November 15, 2026:
| Maker | Current models | How the maker describes them | Context window |
|---|---|---|---|
| Anthropic | Claude Fable 5.2, Opus 5.5, Sonnet 5.5, Haiku 5.5 | Fable 5.2 for demanding reasoning, Opus 5.5 for long-running coding and knowledge work, Haiku 5.5 for high-volume tasks | 1M tokens |
| OpenAI | GPT-6 Astra, GPT-6.1 Sol, GPT-6.1 Terra, GPT-6 Luna | Astra is its most capable model; Luna is its most efficient, for high-volume tasks | 1.05M tokens |
| Gemini 4 Carbon, Argon, Hydrogen | Carbon is the strongest reasoner; Argon is faster; Hydrogen is for everyday coding | 1,048,576 tokens | |
| Google (open) | Gemma 4 (E2B, E4B, 12B, 26B, 31B) | Its most capable open models, sized for hardware from cloud servers to phones | Not stated on the overview page |
The first three are closed, reachable only through their makers' apps and APIs. Gemma is open-weight: Google lists it on Kaggle, Hugging Face, Ollama and LM Studio for developers to run themselves.
Where LLMs go wrong: hallucination
Anthropic's guide to reducing hallucinations defines hallucination as text that is factually wrong or inconsistent with the given context, and says even the most advanced models produce it.
It follows from the training. Pretraining rewards likely text, and a fluent wrong answer can be very likely. The GPT-3 authors noted that their models are not grounded in the physical world.
Human feedback helps without curing it. On tasks where the answer should come only from the input, such as summarizing a document, InstructGPT made up information 21% of the time, against 41% for GPT-3, and its authors still list making up facts among its mistakes. Anthropic tells developers to let the model say it doesn't know and to make it quote and cite its sources, which reduces hallucinations without eliminating them. Our AI hallucination explainer goes further.
What this means for using one
An LLM learned what good text looks like, not what is true. It is strongest when you hand it the facts, the document or the code, and weakest when it has to recall a precise detail on its own. Check names, numbers and citations before you rely on them.
Frequently asked questions
What is an LLM token?
A token is the chunk of text an LLM reads and writes, often a word or part of one. Google's Gemini docs put it at about 4 characters, with 100 tokens equal to roughly 60 to 80 English words.
What is an LLM good for?
The GPT-3 paper showed a single pretrained model handling translation, question answering and fill-in-the-blank tasks from a few examples in the prompt. Current models from Anthropic, OpenAI and Google are pitched for coding, reasoning and agent work, but they still make mistakes that need checking.
What is an open source LLM?
Usually it means an open-weight model whose parameters you can download and run yourself, such as Google's Gemma 4, which Google lists on Kaggle, Hugging Face, Ollama and LM Studio. Closed models such as Claude, GPT-6 and Gemini 4 are available only through their makers' apps and APIs.
What does LLM mean in AI?
LLM stands for large language model, a transformer network with billions of parameters trained to predict the next token in text. It is unrelated to the LL.M., a law degree.
Sources
- Attention Is All You Need (arXiv:1706.03762)arXiv / Vaswani et al., Google
- Scaling Laws for Neural Language Models (arXiv:2001.08361)arXiv / Kaplan et al., OpenAI
- Language Models are Few-Shot Learners (arXiv:2005.14165)arXiv / Brown et al., OpenAI
- Training language models to follow instructions with human feedback (arXiv:2203.02155)arXiv / Ouyang et al., OpenAI
- Context windowsAnthropic
- Models overviewAnthropic
- Reduce hallucinationsAnthropic
- ModelsOpenAI
- Understand and count tokensGoogle
- Gemini modelsGoogle
- GemmaGoogle DeepMind
