Almost every frontier model launch says the system was red teamed. The phrase is cheap to write and expensive to interpret, because it covers anything from an afternoon of prompt poking to a months-long campaign with domain experts and privileged model access. This guide uses seven primary documents (a NIST profile, four research papers, an OpenAI white paper and a published critique) to define AI red teaming, separate its manual and automated forms, and end with questions you can ask of any red-team claim, including the ones in a system card.
What red teaming means
NIST's generative AI profile (NIST AI 600-1, July 2024) lists AI red-teaming alongside participatory engagement and field testing as a form of structured public feedback. It defines it as "a structured testing exercise used to probe an AI system to find flaws and vulnerabilities such as inaccurate, harmful, or discriminatory outputs, often in a controlled environment and in collaboration with system developers."[1]
The same document is careful about what the exercise needs. It says the quality of red-teaming output is related to the background and expertise of the team, that diverse teams can identify flaws in different deployment contexts, and that results "should be given additional analysis before they are incorporated into organizational governance and decision making."[1] It names four team types:
- General public: ordinary users who bring their own experience, often more effective with large groups.
- Expert: specialists in the domain, such as medicine, biotech or cybersecurity.
- Combination: experts verifying or modifying prompts written by the public, useful when specialists are hard to recruit.
- Human / AI: generative AI working with human teams, which NIST says can be more cost effective than humans alone and may suit eliciting different harms.[1]
NIST also says its section focuses on red-teaming in pre-deployment contexts, and that red-teaming can happen before or after release.[1] That is a useful reminder that "was red teamed" says nothing yet about when.
Manual red teaming: what a large study found
The most detailed public account of manual red teaming at scale is Anthropic's 2022 paper. It collected 38,961 red-team attacks and released them as a dataset. The red team was 324 US-based crowdworkers, 307 recruited through Amazon Mechanical Turk and 17 through Upwork, testing models of 2.7B, 13B and 52B parameters across four types: a plain language model, a model prompted to be helpful, honest and harmless, a rejection-sampling model, and one trained with reinforcement learning from human feedback (RLHF).[2]
Its headline finding concerns scale: "the RLHF models are increasingly difficult to red team as they scale", while the other three model types showed a flat trend.[2] That is a result about those models in 2022, under that protocol, and the authors' own discussion shows why not to stretch it.
- Raters disagree. When the authors and reviewers rated attack success on a 0 to 4 scale, they report Fleiss's kappa of 0.32; binarising the rating (success if 3 or above) raised agreement to 0.49.[2]
- Some work was low quality. The authors observed template attacks, such as asking for an insulting term for a person that starts with a given letter, which let a worker generate many attempts without careful thought; they note crowdworkers are incentivised to find such shortcuts.[2]
- Expertise gaps. Some attacks needed domain knowledge to judge, for example weapons or chemistry questions, and the authors were unsure whether the workers had it. They suggest asking domain specialist organisations to red team.[2]
- The harm space is open. They write that because language models are general purpose and open-ended, "the space of possible harms is unknown and unbounded."[2]
Read together, these are the limits of any crowd-scale manual exercise: cost, rater noise and uneven expertise. They are also why the authors flagged automation as the next step.
Automated red teaming: from classifiers to benchmarks
DeepMind's 2022 paper by Perez and colleagues proposed generating test cases with another language model and scoring the target's replies with a classifier trained to detect offensive content. Applied to a 280B-parameter chatbot, it found what the authors describe as tens of thousands of offensive replies. It compared approaches from zero-shot generation to reinforcement learning, and used prompt engineering to surface other harms, including leaked training data, invented contact details and problems that emerge over a conversation. The authors call the method a promising tool, one of many needed.[3]
Automation introduced a new problem: results across papers were hard to compare. HarmBench, a 2024 standardised framework, responds by comparing 18 red-teaming methods against 33 target models and defenses under common conditions. Its authors report that "no current attack or defense is uniformly effective, and that robustness is independent of model size."[4] Two practical consequences follow. A defence that blocks one attack family has not been shown to block the others, and a larger model is not automatically harder to break.
A third paper from OpenAI researchers addresses a trade-off in automated attackers. In the authors' account, methods tend to optimise either diversity or effectiveness and rarely both; they split the task into generating diverse attack goals and then generating effective attacks for each, using reinforcement learning, and report more diverse and effective attacks on prompt injection and unsafe-response elicitation than prior general approaches.[5] That is the authors' own comparison; we have not reproduced it.
How labs describe their own campaigns
OpenAI's white paper on external red teaming is a rare description of process from a developer. It separates manual (humans craft prompts and judge outputs), automated (models or templates generate prompts, sometimes with classifiers grading) and mixed methods, and says each may serve different purposes and are not mutually exclusive.[6]
Three design choices in the paper explain why two "red-team results" can differ:
- Who tests. The paper says the ideal team composition varies by background, education, location and language. For GPT-4 it says OpenAI prioritised natural sciences, autonomous capabilities and power-seeking behaviour, and cybersecurity; for DALL-E 3, misinformation, bias and representation.[6]
- What they can access. Early snapshots can reveal pre-mitigation capabilities, but may produce findings that are "not actionable because the results will be obsolete with the updated system"; once mitigations are enabled, testing can turn to residual risks and how robust the mitigations are.[6]
- What happens next. Human red-team prompts can seed repeatable automated evaluations. The paper's example is DALL-E 3, where open-ended red teaming found gaps in misinformation-prone images, explicit-content jailbreaks and self-harm imagery, and prompts from testers seeded automated evaluations for selected subdomains.[6]
The white paper is also direct about limits. It lists relevance across evolving models, resource intensity, harms to participants, information hazards from disclosing jailbreaks, "picking early winners" among testers with early access, and the rising threshold of human sophistication as models improve. It adds that findings from one point in time "may be under-assessed or no longer reflected in an updated system."[6]
The critique: a result needs a purpose
Feffer and co-authors surveyed red-teaming activity in industry and the research literature. They found practices diverge on the purpose of the activity (often vague), the artifact being evaluated, the setting (actors, resources, methods) and the decisions the results inform, such as reporting, disclosure or mitigation. Their conclusion: red teaming may be a valuable "big-tent idea" and industry may apply it effectively behind closed doors, but gestures toward it, based on public definitions, as a panacea for every risk "verge on security theater." They offer a question bank to scaffold future practice.[7]
This does not say red teaming is useless. It says the label does not specify the test.
How to read a red-team claim
Our interpretation, built from the sources above rather than quoted from any one of them, is that a claim becomes informative when it answers these questions:
| Question | Why it matters | Source of the concern |
|---|---|---|
| Who tested, and with what expertise? | Quality tracks team background and domain knowledge | NIST AI 600-1[1] |
| What did they have access to? | Pre-mitigation and post-mitigation testing answer different questions | OpenAI white paper[6] |
| Which version was tested? | Findings can be stale after an update | OpenAI white paper[6] |
| How was success judged, and how consistently? | Rater agreement was moderate even in a careful study | Anthropic 2022[2] |
| Manual, automated or both? | Automated sweeps scale, human testers find novel problems | OpenAI, Perez et al.[3][6] |
| Which attack families were covered? | No attack or defence was uniformly effective | HarmBench[4] |
| What decision did the result inform? | A result with no consequence is not a safeguard | Feffer et al.[7] |
If a launch post answers few of these, treat "red teamed" as a statement that testing took place, not as evidence about outcomes. For where red teaming sits among a developer's other safeguards, see our comparison of frontier safety frameworks, the NIST AI RMF, and for outside evaluators with deeper access, embedded AI evaluators. If you deploy agents, red teaming for prompt injection is a separate discipline; start with how AI agents work and how to secure them.
What we did not verify
We read each source's page or full text as of 7 November 2026 (the NIST profile, the Anthropic, HarmBench, Feffer and OpenAI white-paper PDFs; for the Perez and Beutel papers we read the abstract pages only). We ran no red-team tests ourselves, and nothing here measures any current model. The studies date from 2022 to 2025, so their numbers describe the models and methods of those years.
