AI Red Teaming for generative AI systems

Classic red teaming simulates attackers against networks, systems and identities – the findings are usually clearly reproducible technical vulnerabilities. AI red teaming, by contrast, targets the behaviour of the AI system itself: language models respond probabilistically, the same attack may only succeed on the tenth attempt, and a finding is often not a programming error but undesirable model behaviour. Beyond classic security objectives, testing also covers content-related harms, such as the generation of harmful or false content. Equally important is the distinction from benchmarks: static test datasets measure known capabilities, while red teaming deliberately hunts for novel, context-specific failure modes – one of the central lessons of the Microsoft AI Red Team from more than 100 GenAI products tested.

Four families of techniques take centre stage. Jailbreaks defeat a model's safety guardrails to unlock restricted content or functions. In direct prompt injection, the attacker places manipulative instructions in their own input; in indirect prompt injection, they hide them in content the system processes – such as web pages, documents or emails. Data extraction targets system prompts, training data or connected knowledge sources – listed in the OWASP Top 10 for LLM Applications 2025 as Sensitive Information Disclosure (LLM02) and System Prompt Leakage (LLM07), while Prompt Injection tops the list as LLM01. Particularly critical is unsafe tool use (Excessive Agency, LLM06): if a model can send emails, query databases or execute code, an injected instruction becomes an executed action.

With the GenAI Red Teaming Guide (version 1.0, January 2025), the OWASP GenAI Security Project has published a structured, vendor-neutral methodology. The guide covers four testing layers: evaluation of the model itself, the implementation (including guardrails and system prompts), the surrounding infrastructure, and runtime behaviour in interaction with users and processes. Complementing it, NIST AI 100-2 E2025 (March 2025) provides a taxonomy and terminology for adversarial machine learning that classifies attacks on predictive and generative AI systems by the attacker's objectives, capabilities and lifecycle phase. Together, the two documents form the vocabulary needed to define test scope and assessment criteria in a traceable way.

An AI red teaming engagement begins with scoping: which use case, which data classes, which connected tools, what potential for harm? This feeds into threat modelling with concrete attack scenarios, prioritised by realistic impact rather than mere feasibility. Testing combines both approaches: automated with frameworks such as Microsoft's open-source PyRIT for breadth of coverage, and manual for context-specific attack chains – in the experience of the Microsoft AI Red Team, automation extends coverage but does not replace human expertise. An assessment grid classifies findings by severity, reproducibility and the security objectives affected. The report documents each finding with traceable evidence (prompts and model responses) and derives hardening measures, such as more restrictive tool permissions, input and output filters, or revised system prompts.

The AI Act (Regulation (EU) 2024/1689) makes adversarial testing an explicit legal obligation for the first time: under Art. 55(1)(a), providers of general-purpose AI models with systemic risk must perform model evaluations in accordance with standardised protocols and state-of-the-art tools – including conducting and documenting adversarial testing to identify and mitigate systemic risks. These obligations have applied since 2 August 2025. They directly affect only a small number of model providers, but they set the benchmark for the entire supply chain. For high-risk AI systems, Art. 15(5) additionally requires resilience against AI-specific attacks – explicitly naming, among others, data poisoning, model poisoning, adversarial examples or model evasion, and confidentiality attacks. AI red teaming is an obvious way to demonstrate the effectiveness of such measures.

A point-in-time assessment – for example before go-live or as the basis for a release decision – delivers a reliable snapshot. It only remains valid as long as the system does not change: model updates from the provider, adjusted system prompts, new tools and data sources, or new attack techniques continuously shift the risk picture. The Microsoft AI Red Team draws the lesson that securing AI systems is never finished. For production AI applications that keep evolving, a continuous approach is therefore the natural fit – with automated regression tests for every relevant change, periodic manual deep-dive testing, and firm anchoring in the AI governance and development process.