UserArena benchmark tests AI shoppers against 11,010 real users

UserArena is a benchmark for testing how well an AI can pretend to be a shopper. A team from Leiden University, Renmin University of China, Alibaba Group and the Chinese Academy of Sciences posted it to arXiv on October 22. It pairs 64,729 real shopping sessions from 11,010 users with an interactive environment, and it checks simulators against the result of a real production A/B test.

The point is that companies now build agents that make long chains of shopping decisions, and they need a stand-in for customers to train and test them on. If the stand-in behaves unlike real people, every conclusion drawn from it is suspect. UserArena is an attempt to measure that gap.

The problem: testing shopping agents without real shoppers

Trying a new ranking or recommendation policy on live users is slow and costly. A user simulator is a model that plays the shopper: it sees a page, then clicks, scrolls, searches or buys. The authors say existing datasets rarely have long, fine-grained real-user trajectories, rarely score whole trajectories instead of single steps, and rarely check simulated results against a production A/B test.

What is in the benchmark

The data comes from "one of Asia's largest e-commerce platforms", which the paper does not name. The authors are partly from Alibaba, and the work was supported by an Alibaba research program. The paper reports:

• 11,010 users, 64,729 session trajectories and 577,670 actions
• 566,102 products across 41 categories, with apparel making up 54.90% of the 73,717 item clicks
• 13 action types
• 59,378 sessions from 5,768 users that are linked into cross-session histories, averaging 10.29 sessions and 86.36 actions per user

Each step lines up the user profile, what the page looked like, the product context and the action taken. The pages are rebuilt in an offline environment from de-identified logs, so the released images are not real screenshots.

Simulators are scored at five levels. Level 0 checks that the model outputs a valid action. Level 1 checks whether it picks the same next action as the real user. Level 2 checks whether a free-running simulation has realistic dynamics, such as how long sessions last. Level 3 checks whether it matches different groups of users. Level 4 checks outcomes: errors in clicks per feed action, add-to-cart actions per click, and how sessions end.

The authors' own method, OPTD

The team also trained a simulator of its own, called On-Policy Trajectory Distillation (OPTD). A large model first writes a short summary of what the user was trying to do across a whole trajectory. A frozen teacher model sees that summary, and the student model, which never sees it, is trained to match the teacher's choices on its own generated steps. A bound on how far the teacher's outputs may drift keeps the summary from leaking the answer.

Results: OPTD leads, with exceptions

Every method in the main comparison used the same 2-billion-parameter Qwen3.5 base model, so differences come from the method and not model size. Eight simulators from earlier work were tested against OPTD: RecAgent, Agent4Rec, SimUSER, Shop-R1, Customer-R1, AlignUSER, A/B Agent and STA.

In single-session tests, OPTD scored best on all eight Level 1 to 4 metrics. Its macro-F1 was 0.4178 against 0.4108 for Customer-R1, and its trajectory-length error was 1.72 against 1.77 for STA. Its click-rate error was 0.0026 (lower is better), against 0.0227 for Customer-R1.

Cross-session results were less clean. OPTD was first on five of the eight metrics and second on two more. STA beat it on macro-F1 (0.3556 against 0.3456), and A/B Agent had a lower add-to-cart error, though its share of valid actions was far lower (0.0096 against 0.1399). The authors note that gains are not consistent across levels.

The check against a real A/B test

The test the authors stress most compares one pair of ranking policies in simulation, using 10,000 users split evenly, against the change recorded in a production A/B test. The measure is the effect ratio: the simulated change in click-through rate divided by the real one, where 1.0 is perfect.

OPTD came closest, but it still understated the real effect by 28.7%. The two general-purpose GPT-6 models overshot it, which the authors read as a sign that a bigger model does not give realistic sensitivity to ranking changes without task-specific tuning. The other fine-tuned simulators mostly flattened the effect.

Do humans notice the difference

In a blinded test, three annotators looked at 200 pairs per simulator, each with one real and one simulated trajectory, and picked the machine-made one or said unsure. Annotators identified the simulated trajectory correctly 38.7% of the time for OPTD, the lowest among the four simulators tested, and mistook a real trajectory for the simulated one 26.3% of the time, the highest. Agreement among annotators was moderate (Fleiss' kappa 0.528). Dropping any OPTD component hurt at least one metric, and dropping the supervised warm-up hurt most.

Limits

• The production comparison uses one pair of ranking policies. One match, even a good one, does not show the benchmark predicts A/B outcomes in general.
• OPTD's effect ratio is 0.713, not 1.0. The paper says every simulator kept some gap.
• Everything in the main table uses one small Qwen3.5 model of about 2 billion parameters, so the results say nothing about larger models trained the same way. The GPT-6 models appear only in the A/B check.
• The human test covers four simulators and three annotators, and the data comes from a single platform, mostly apparel.
• The authors designed the benchmark and the winning method, so the leading result is theirs. The paper has no independent replication yet.

Code and data

The GitHub repository has the simulator, evaluation tools and the OPTD training pieces. The released data is a held-out set of 535 sessions, not the full 64,729, and no trained models are included. Use is restricted to non-commercial research under a data use agreement that bars re-identifying users and redistribution. The authors say a leaderboard is planned.