Key takeaways

  • NIST defines AI red teaming as a structured testing exercise to find flaws such as inaccurate, harmful or discriminatory outputs in an AI system.
  • It differs from a pentest because it also tests for harmful output from a system working as built, and because the same prompt can give different answers each time.
  • Labs combine human red teamers with automated tools such as Microsoft's PyRIT, but red teaming can show that problems exist, not that none are left.

AI red teaming is the practice of attacking an AI system on purpose, before real attackers or ordinary users do, to find the ways it can fail or be misused. NIST's generative AI profile (NIST AI 600-1), published in July 2024, defines it as "a structured testing exercise used to probe an AI system to find flaws and vulnerabilities," such as inaccurate, harmful or discriminatory outputs, usually in a controlled setting and together with the system's developers.

The red team plays the adversary. The developer then fixes what broke and tests again.

What AI red teaming looks for

The term comes from Cold War war games, where the "red" side played the Soviet Union, and security teams later borrowed it for simulated hacking. With AI, the list of failures is longer. NIST's profile tells organizations to red team for, among other things:

  • abuse of the model to attack other systems, such as writing malicious code or better phishing emails
  • attacks on the AI itself, such as prompt injection, data poisoning and model extraction
  • leaks of training data or personal, confidential or copyrighted information
  • biased answers about groups of people
  • dangerous, violent or hateful content

Some also test for confident wrong answers, or AI hallucination. As models turn into AI agents that browse, run code and move money, the target grows from what a model says to what it does.

How it differs from a security pentest

A penetration test checks whether an attacker can break into a system: steal data, take over an account, escalate access. AI red teaming includes that, but it adds a second question: can the system be talked into doing harm while it works exactly as built?

Microsoft named three differences when it released PyRIT, its red teaming toolkit, on February 22, 2024:

  1. Two kinds of risk at once. It looks for security holes and for harms such as unfair or false output.
  2. Randomness. The same prompt can get a different answer each time, so one failed attack proves little. A pentester can usually repeat an exploit and get the same result.
  3. Many shapes of system. A model can be a chat app, a feature inside another product, or something that reads images, audio and video.

Google's July 2023 AI Red Team report adds that most attacks on real AI systems will be "run-of-the-mill" cyber threats, so its AI red team works alongside Google's traditional red teams.

How it differs from jailbreaking

A jailbreak is one technique. Anthropic defines it as an input designed to get around a model's safety guardrails and make it produce harmful responses. Red teaming is the whole exercise: deciding what to test, who tests it, trying many techniques, jailbreaks included, measuring how often they work, and feeding the results back into fixes.

Microsoft's AI Red Team, writing in January 2025 about lessons from red teaming over 100 generative AI products, says simple tricks usually beat clever math. Hand-written jailbreaks and multi-turn conversations work as well as attacks that need full access to the model's internals, and they are the attacks that actually circulate. In one case, a vision model refused a harmful request typed as text but often complied when the same words were placed inside an image.

The same paper draws a second line: red teaming is not safety benchmarking. A benchmark measures known harms with a fixed test set. Red teaming goes looking for harms nobody has written a test for yet. Its findings can become the next benchmark.

Manual vs automated red teaming

OpenAI's white paper on external red teaming, released in November 2024, splits the methods three ways:

Method How it works Good at
Manual People write prompts and interact with the model to act out adversarial scenarios New risks, expert domains, judging subtle harm
Automated A model or templates generate attack inputs, and classifiers often grade the outputs Scale, repeat runs, measuring how often a failure happens
Mixed Humans create a seed set of attacks, then automation scales it up Turning one-off findings into tests that can be rerun

Automated red teaming started in research papers. In Red Teaming Language Models with Language Models, posted in February 2022, Ethan Perez and colleagues used one language model to write test questions for another, a 280-billion-parameter chatbot, and a classifier to flag offensive replies. It found tens of thousands, plus leaked phone numbers and training data.

Manual work at scale came from Anthropic. In a paper posted in August 2022, Deep Ganguli and colleagues red teamed models of 2.7, 13 and 52 billion parameters and released 38,961 human-written attacks. Models trained with RLHF, reinforcement learning from human feedback, got harder to break as they grew. The other model types did not.

Labs now run both. In February 2025 Anthropic tested its Constitutional Classifiers with 10,000 synthetic jailbreak prompts: 86% got through without the classifiers and 4.4% with them. It also paid human red teamers, 183 active participants, who spent an estimated 3,000-plus hours over two months trying for a $15,000 reward. None found a universal jailbreak.

Who does AI red teaming

  • The labs' own teams. Google, Microsoft, OpenAI and Anthropic all run internal red teams. Anthropic's June 2024 overview lists domain experts on child safety and elections, national security testing for biological, nuclear and cyber risks, multilingual, automated and multimodal testing, and public events such as DEF CON's AI Village.
  • Outside experts. OpenAI says it has used external red teamers on its frontier launches since DALL-E 2 in 2022. Its paper credits them with finding that GPT-4o's voice mode could unintentionally imitate the user's voice.
  • Governments. The UK AI Security Institute says its red team has spent over two years breaking misuse safeguards so developers can fix them, including work with Google DeepMind and Anthropic. A public agent red teaming competition it reported on in July 2025 collected about 1.8 million prompt injection attacks on 22 AI agents. More than 60,000 caused policy violations, and nearly all agents broke within 10 to 100 queries.

AI red teaming tools

  • PyRIT (Microsoft, MIT license): generates attack prompts, scores the replies, adapts the next attack, and runs multi-turn attacks. Microsoft says it is not a replacement for manual red teaming.
  • garak (NVIDIA, Apache-2.0): a scanner that probes a model for jailbreaks, prompt injection, data leakage, toxic output and hallucination.
  • Promptfoo (open source): generates adversarial inputs from "plugins" for failure types, including authorization bugs in apps built on a model.

The limits of red teaming

Red teaming finds problems. It cannot prove there are none. OpenAI's paper lists the main limits: results go stale as models are updated, the work is expensive, red teamers are exposed to disturbing content, publishing a new jailbreak can help attackers, and as models get smarter it takes more expertise to judge whether an answer is dangerous.

Anthropic adds that there is no standard practice. Labs test the same threat in different ways, which makes it hard to compare how safe two models are. NIST's own definition calls AI red teaming "an evolving practice."

Microsoft's lessons paper ends the same way: securing AI systems will never be finished. Each round of testing and fixes raises the cost of an attack, and today jailbreaks are still cheap.

Frequently asked questions

What does AI red teaming mean?

It means attacking an AI system on purpose to find how it can fail or be misused before release. NIST calls it a structured testing exercise to find flaws such as inaccurate, harmful or discriminatory outputs.

What are the main AI red teaming tools?

Widely used open-source tools include Microsoft's PyRIT, NVIDIA's garak scanner and Promptfoo's red team feature. They generate attack prompts and score the replies, and Microsoft says such tools support manual red teaming rather than replace it.

Is there an AI red teaming certification standard?

Not a shared one from the labs. In June 2024 Anthropic said red teaming lacks standard practices and asked policymakers to encourage a certification process based on common standards, with NIST developing them.

How does red teaming work for AI agents?

Testers try to make an agent take harmful actions, often through prompt injection hidden in content it reads. In a public competition reported by the UK AI Security Institute in July 2025, nearly all 22 agents tested broke within 10 to 100 queries.

Sources

  1. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1)NIST
  2. Red Teaming Language Models with Language ModelsarXiv (Perez et al.)
  3. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons LearnedarXiv (Ganguli et al., Anthropic)
  4. OpenAI's Approach to External Red Teaming for AI Models and SystemsOpenAI
  5. Challenges in red teaming AI systemsAnthropic
  6. Constitutional ClassifiersAnthropic
  7. Why Red Teams Play a Central Role in Helping Organizations Secure AI SystemsGoogle
  8. Announcing Microsoft's open automation framework to red team generative AI systemsMicrosoft
  9. Lessons From Red Teaming 100 Generative AI ProductsarXiv (Bullwinkel et al., Microsoft)
  10. PyRITMicrosoft
  11. How our new Control Red Team is stress testing frontier monitorsUK AI Security Institute
  12. Security challenges in AI agent deployment: insights from a large-scale public competitionUK AI Security Institute
  13. garakNVIDIA
  14. LLM red teamingPromptfoo

Add ClaudeGPT Si as a preferred source on Google