# What is GRPO: how group relative policy optimization trains LLMs

> GRPO scores a group of answers to the same prompt against their average, which drops the value model PPO needs and made RL on DeepSeek-R1 cheaper.

URL: https://claudegpt.si/news/what-is-grpo-group-relative-policy-optimization  
Published: 2026-11-20T11:00:35.646Z  
Author: ClaudeGPT Si Newsroom  
Section: Research  
Topics: Reinforcement learning, Reasoning models, Research, Language models, DeepSeek, Fine-tuning  
Publisher: ClaudeGPT Si

## Key takeaways

- GRPO samples a group of answers per prompt and scores each against the group's mean and standard deviation, so it needs no separate value model.
- DeepSeek introduced it in DeepSeekMath and used it for DeepSeek-R1-Zero, whose AIME 2024 pass@1 rose from 15.6% to 77.9% during training.
- Follow-ups such as Dr. GRPO and DAPO fix length and difficulty biases in the original loss, and TRL's GRPOTrainer now defaults to the DAPO loss.

GRPO, short for group relative policy optimization, is a reinforcement learning method for training language models. The model answers the same prompt several times, each answer is scored, and each answer is then judged against the group's average: better than average gets pushed up, worse gets pushed down. DeepSeek introduced it in its [DeepSeekMath paper](https://arxiv.org/abs/2402.03300), and it became widely known as the training method behind [DeepSeek-R1](https://arxiv.org/abs/2501.12948).

The point of the trick is cost. GRPO gets rid of the second large model that the older standard method, PPO, needs during training. That made reinforcement learning on big models cheaper to run, and it is why GRPO now shows up in open-source training libraries and in Google's own tutorials.

## The problem GRPO solves

[Reinforcement learning](https://claudegpt.si/news/what-is-reinforcement-learning) teaches a model by rewarding good outputs instead of showing it correct ones. For language models it is part of [post-training](https://claudegpt.si/news/what-is-post-training), the stage after pretraining, and it is the idea behind [RLHF](https://claudegpt.si/news/what-is-rlhf).

The workhorse algorithm was [PPO (proximal policy optimization)](https://arxiv.org/abs/1707.06347), published by John Schulman and colleagues in July 2017. PPO needs a baseline to know whether a reward is good or bad, and it gets one from a value model: a second network that predicts how much reward the current answer is likely to earn. The DeepSeekMath authors point out that this value model is usually about as large as the model being trained. Training a 7-billion-parameter model with PPO means training a second network of similar size alongside it, with the memory and compute that come with it.

## How GRPO works

GRPO replaces the value model with a simple comparison inside a group:

1. Take one question and sample a group of answers from the current model. DeepSeekMath sampled 64 per question.
2. Score each answer with a reward. That can be a reward model, or a rule such as "is the final number correct".
3. Compute each answer's advantage: its reward minus the group's mean, divided by the group's standard deviation.
4. Update the model so answers with a positive advantage become more likely and the rest less likely, using a clipped objective like PPO's that stops any single update from moving the model too far.

There is no learned critic. The other answers in the group are the baseline. GRPO also moves the KL penalty, a term that keeps the model close to a frozen reference copy of itself, out of the reward and into the loss directly.

With outcome rewards, every token in an answer gets that answer's advantage. The paper also describes a process version, where each reasoning step is scored separately.

## What it did for DeepSeekMath and DeepSeek-R1

In DeepSeekMath, GRPO turned DeepSeekMath-Instruct 7B into DeepSeekMath-RL 7B using about 144,000 math questions from GSM8K and MATH. Accuracy rose from 82.9% to 88.2% on GSM8K and from 46.8% to 51.7% on MATH, with chain-of-thought prompting.

The bigger demonstration came with DeepSeek-R1. For R1-Zero, DeepSeek applied GRPO directly to its DeepSeek-V3-Base model, skipping the usual supervised fine-tuning step. The rewards were simple rules: one for a correct answer, one for putting the reasoning inside `<think>` tags. The paper reports that average pass@1 on the AIME 2024 math exam rose from 15.6% to 77.9% during training, and that the model learned on its own to write longer chains of thought and to stop and recheck its work. R1-Zero sampled 16 answers per question and used a KL coefficient of 0.001. Its output was hard to read and mixed English and Chinese, which is why the released R1 added a multi-stage pipeline on top. Our explainer on [how reasoning models work](https://claudegpt.si/news/how-do-reasoning-models-work) covers what came out of that.

## GRPO vs PPO

| | PPO | GRPO |
| --- | --- | --- |
| Baseline | Learned value model | Mean reward of a group of answers to the same prompt |
| Extra network trained | Yes, often as large as the policy | No |
| Answers sampled per prompt | No group needed | A group (64 in DeepSeekMath, 16 in R1-Zero) |
| KL penalty | Added to the reward per token | Added to the loss |

## Where GRPO falls short

Follow-up papers found problems in the original formula. [Understanding R1-Zero-Like Training](https://arxiv.org/abs/2503.20783), a March 2025 paper by Zichen Liu and colleagues, identified two biases. Dividing each answer's loss by its length means long wrong answers are penalized less than short ones, which pushes models toward long incorrect answers. Dividing by the group's standard deviation gives extra weight to questions that are almost always right or almost always wrong. Their fix, Dr. GRPO, removes both terms. With it, a 7B Qwen2.5-Math model reached 43.3% on AIME 2024.

[DAPO](https://arxiv.org/abs/2503.14476), an open-source system published in March 2025, made four changes: a higher upper clipping bound so rare tokens can still grow, dropping prompts where every answer in the group got the same reward (their advantage is zero, so they teach nothing), averaging the loss over tokens instead of answers, and a softer penalty for answers cut off at the length limit. It also removed the KL penalty. On Qwen2.5-32B it scored 50 on AIME 2024, against 47 for DeepSeek's R1-Zero recipe on the same base model, in half the training steps.

There is also a limit on what GRPO teaches. The DeepSeekMath authors found that RL improved Maj@K, the accuracy of a majority vote over K samples, but not Pass@K, whether any of K samples is right. In their words, the gain came from "boosting the correct response from TopK rather than the enhancement of fundamental capabilities." GRPO makes good answers the model could already produce more likely. That is useful, but it is not the same as teaching new skills.

## How people use GRPO now

Hugging Face's [TRL library](https://huggingface.co/docs/trl/grpo_trainer) ships a `GRPOTrainer` that takes a model, a dataset and one or more reward functions. Its defaults show how far practice has moved from the 2024 paper: 8 answers per prompt, the KL term switched off (`beta` 0.0), and a `loss_type` of `dapo`, with `grpo` and `dr_grpo` available as options.

GRPO is also moving from single answers to agents. On November 19, Google published a [tutorial on multi-turn agentic reinforcement learning](https://developers.googleblog.com/multi-turn-agentic-reinforcement-learning-with-tunix-and-gemma-4-on-google-tpus/) with Tunix, its post-training library for Cloud TPUs. It trains Gemma 4 E2B to play Plane Strike, a game on a 6x6 grid where the model hunts for a hidden 8-cell plane where each move is a tool call. The whole game is scored as one trajectory, and the learner runs a GRPO-family setup with RLOO advantages and a GSPO token-level loss. Google reports the base model won 9.4% of games on held-out boards. After 100 training steps on a single TPU v6e-8 host, it won 95%, in about 16 moves instead of 22.44.

That is where GRPO is useful: any task where a program can check the result. Math answers, passing tests and won games all qualify. Where the reward has to come from human taste, a learned reward model is still needed, and the group comparison only helps as much as that model's scores are right.

## Frequently asked questions

### What does GRPO stand for?

GRPO stands for group relative policy optimization. It was introduced by DeepSeek in the DeepSeekMath paper as a variant of PPO.

### What is the difference between GRPO and PPO?

PPO trains a separate value model, often as large as the policy, to estimate a baseline for each answer. GRPO drops that model and uses the mean reward of a group of answers to the same prompt as the baseline.

### What is the GRPO advantage formula?

For each answer in a group, the advantage is its reward minus the group's mean reward, divided by the group's standard deviation. With outcome rewards, every token in the answer gets that value.

### What is Dr. GRPO?

Dr. GRPO, from the paper Understanding R1-Zero-Like Training, removes GRPO's per-answer length normalization and its standard-deviation scaling. The authors found both terms biased training, the first toward long incorrect answers.

## Sources

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://arxiv.org/abs/2402.03300), arXiv (DeepSeek-AI)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://arxiv.org/abs/2501.12948), arXiv (DeepSeek-AI)
- [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347), arXiv
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://arxiv.org/abs/2503.20783), arXiv
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://arxiv.org/abs/2503.14476), arXiv
- [GRPO Trainer](https://huggingface.co/docs/trl/grpo_trainer), Hugging Face
- [Multi-Turn Agentic Reinforcement Learning with Tunix and Gemma 4 on Google TPUs](https://developers.googleblog.com/multi-turn-agentic-reinforcement-learning-with-tunix-and-gemma-4-on-google-tpus/), Google Developers Blog
