# Multiscale Policy Optimization trains LLM agents turn by turn

> A new paper credits an agent's whole turns first, then its tokens, and reports gains of up to 4 points on web search tests over PPO, VAPO and SAO.

URL: https://claudegpt.si/news/multiscale-policy-optimization-llm-agents  
Published: 2026-10-28T18:52:35.358Z  
Author: ClaudeGPT Si Newsroom  
Section: Research  
Topics: Reinforcement learning, AI agents, Hugging Face, Qwen, Research papers  
Publisher: ClaudeGPT Si

## Key takeaways

- MPO assigns credit to an agent's whole turns first, then splits it among tokens, instead of treating every token as a separate decision.
- On Qwen3-4B and Qwen3-30B-A3B, the authors report higher pass@1 than GRPO, GSPO, PPO, VAPO and SAO on BrowseComp-Plus, GAIA and HLE.
- No code or weights were linked as of October 28, and the Claw-Eval and Automation Bench training data are proprietary.

Researchers from Peking University, IQuest Research and other institutions have proposed Multiscale Policy Optimization (MPO), a way to train language-model agents on tasks that take many steps. The [paper](https://arxiv.org/abs/2610.35013) was posted to arXiv on October 26, 2026, and was one of the most upvoted on Hugging Face's daily papers page on October 28. The authors report that MPO beats the reinforcement learning methods in common use on web search, tool-use and automation tests.

The idea is simple to state. When an agent finishes a long task, it gets one score at the end. Training has to work out which of its many steps, and which words inside each step, deserve the credit or the blame. MPO changes where that bookkeeping happens.

## The problem: one score for a hundred decisions

An agent such as a research assistant searches the web, reads a page, searches again, calls a tool and finally writes an answer. Each of those actions is a full message, which the paper calls a turn. The model writes each turn one token (a word or word fragment) at a time.

Reinforcement learning (RL) trains the model by rewarding behavior that led to a good result. With a long task, the reward arrives only at the end, so the training method must spread it backward. This is called credit assignment.

The paper describes two families of existing methods and where they fall short:

- Group methods such as GRPO, DAPO and GSPO give every token in a whole attempt the same credit. They cannot tell a good search query from a bad one inside the same run.
- Token-level methods such as PPO, VAPO and SAO use a second model, a critic, that estimates how well things are going after each token. They treat every token as a separate decision, even though an agent really acts in whole turns.

## What MPO does

The authors checked how the critic's estimates behave in one search-agent training run, covering 2,048 attempts and 16,323 turns. They found that differences between turns explained a median of 89.5% of the variation in the critic's values within an attempt. In other words, the value of a situation changes mostly when a turn ends, not from token to token.

So MPO works at the turn level first. It has two parts:

1. Multiscale Advantage Estimation. Credit is first computed per turn, then divided among the tokens inside that turn. The turn's average credit stays fixed, and a smaller adjustment rewards tokens that mattered more within it.
2. Turn-level importance sampling. Each update checks how much the probability of the whole turn changed, rather than token by token, so training stays stable.

## Results

The team trained two Qwen3 models: Qwen3-4B-Thinking and Qwen3-30B-A3B-Instruct, a mixture-of-experts model that uses about 3 billion of its 30 billion parameters at a time. They tested on web search (BrowseComp-Plus, GAIA and HLE), Claw-Eval and Automation Bench. They compared MPO with GRPO, GSPO, PPO, VAPO and SAO.

All figures below are pass@1, the share of tasks solved on the first try, as reported by the authors.

| Benchmark | Model | MPO | Best other method |
| --- | --- | --- | --- |
| BrowseComp-Plus | 4B | 48.08 | 43.75 (SAO) |
| GAIA | 4B | 39.47 | 35.62 (SAO) |
| HLE | 4B | 9.92 | 9.26 (GSPO) |
| BrowseComp-Plus | 30B-A3B | 50.32 | 46.73 (VAPO) |
| GAIA | 30B-A3B | 43.42 | 39.62 (SAO) |
| HLE | 30B-A3B | 10.03 | 9.91 (SAO) |

On Automation Bench, MPO scored 13.33 with the 4B model and 15.00 with the 30B-A3B model, against 12.45 and 14.20 for the strongest comparison. The gains on search are several points. On HLE and Automation Bench they are under one point.

Two tests point to what matters. Replacing turn-level ratios with token-level ones cut search pass@1 by about 2 to 3 points. Removing the within-turn allocation cut it by about 2 to 6 points on the 4B model.

## What to keep in mind

This is a single preprint, and the evidence has gaps the paper itself shows:

- Each training run and setting appears to have been run once, so the paper does not measure how much results vary from run to run.
- The ablation studies were run only on the 4B model on search tasks.
- Only two Qwen3 models were tested.
- For Claw-Eval and Automation Bench, the authors write that the training data are "proprietary internal company datasets," so outsiders cannot reproduce those runs.
- The theory gives worst-case bounds. The authors say these "compare certified worst-case penalties, not actual biases or achieved improvements."

The paper is also silent on code. The arXiv text I read has no code or weights release statement, and the Hugging Face page links no repository, model or dataset. That may change, but as of October 28 there is nothing to download.

## Why it matters

Most of the agents people use today are built from models trained to answer one prompt, then pointed at long tasks. Training directly on long tasks is harder, and the choice of credit assignment is one of the main things that decides whether it works. MPO's claim is that matching the training signal to the unit an agent acts in, the turn, is both simpler and better. The gains are modest on the hardest benchmark and larger on the search tasks, and they need independent replication.

What to watch next is whether the authors release code, and whether other groups reproduce the gains on open data and larger models.

Read the paper on [arXiv](https://arxiv.org/abs/2610.35013) or on [Hugging Face](https://huggingface.co/papers/2610.35013).

## Frequently asked questions

### How do you train LLM agents for long tasks?

Most teams use reinforcement learning, rewarding the model when a multi-step task ends well. The hard part is credit assignment: working out which steps earned the reward. MPO does this per turn first, then per token.

### What is credit assignment in reinforcement learning?

It is the problem of deciding which earlier actions caused a reward that arrives later. For an agent that gets one score at the end of a long task, training must spread that score back over many turns and tokens.

### Is the code for Multiscale Policy Optimization available?

Not as far as the sources show. The arXiv text has no release statement, and the Hugging Face page links no code, models or datasets as of October 28, 2026.

## Sources

- [Multiscale Policy Optimization for Long-Horizon LLM Agents (arXiv)](https://arxiv.org/abs/2610.35013), arXiv
- [Multiscale Policy Optimization for Long-Horizon LLM Agents (full text)](https://arxiv.org/html/2610.35013), arXiv
- [Multiscale Policy Optimization for Long-Horizon LLM Agents](https://huggingface.co/papers/2610.35013), Hugging Face