AI Red Teaming and Emulation Tools

All courses / Tools / Module 4 / Chapter 11

Learning objectives

After the lesson the participant can:

Draft the scope, stop conditions and stop path of a rules-of-engagement manifest for an exercise.
Explain why an expectation per step is written before an emulation run.
Classify the result of an emulated step with the outcome taxonomy of AI Purple.
Assess what open libraries and breach-and-attack simulation do and do not show about a defence.

Key concepts

Rules-of-engagement manifest: A signed, versioned manifest of scope, window, permitted tactics or scenarios, AI participation, stop conditions and stop path for an exercise. It works as a control only when a component outside the executor enforces it.
Emulation plan: A plan that turns signed rules into steps. In AI Red Team each step names an ATT&CK technique, an executor, an autonomy level and the detection or control it expects; in AI Purple the plan, family EMU, sequences three to eight scenarios.
Continuous emulation pipeline definition: An artefact of AI Purple that defines triggers, gates, variation, tagging and the abort path. In the template, safety class S1 inside scope runs at Level 3 Bounded, anything above S1 at Level 2 Supervised, and Level 4 never.
AI red-team campaign plan: A plan for a time-boxed campaign against a copy of an AI system, mapped to OWASP and ATLAS categories. Each objective states a property that must hold, and adversarial inputs are kept out of every record.
Outcome taxonomy: A seven-class classification from AI Purple of what happened to each emulated step. Each step ends in exactly one outcome, the last layer at which the defence did what the expectation said, such as prevented, detected or alerted but not triaged.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Interactive exercise 1 · Put in order

A campaign on a retrieval-augmented assistant

Put the six steps of the practical guide of AI Red Team, Chapter 13, in order. The campaign is authorised, scoped and ends in defensive tests.

Interactive exercise 2 · Match

When each testing tool is the right choice

Match each tool or framework of Chapter 11 with the situation in which its entry says to use it.

Teaching plan

15 min	Present Figure 11.1, an authorised test from scope to finding, and the six questions a manifest answers. Note that the stop path and the policy decision point sit outside the executing system.
20 min	Study the Hansa Mutual emulation plan example: the detection fired but the alert was queued at medium severity and triaged after 47 minutes. Classify the outcome together.
15 min	Teach continuous emulation and AI red teaming: schedule triggers, Atomic Red Team and Caldera, the limit of simulation products, and the Nordvik Region campaign finding of a withdrawn page still in the retrieval index.
30 min	Exercise: groups draft the governing parts of a rules-of-engagement manifest and three expectations for a tabletop scenario.
10 min	Debrief: who can stop each group's exercise, from where, and what defensive change each expected finding would lead to.

Discussion

The same identifier, RoE-NR-2027-001, names a shorter manifest in AI Red Team and a longer one in AI Purple. What should a team that adopts the template do about version and provenance?
The practical guide classifies every AI component that executes emulation steps as Mode 5 with its own permission manifest. What risk does that classification address?
A detection that fires but is triaged after 47 minutes: is that a detection problem or a process problem, and which record should carry the change?

Written exercise

Exercise · 30 minutes

Rules of engagement for a scoped exercise

As a design exercise only, draft the governing parts of a rules-of-engagement manifest for a purple exercise on a system of your own organisation, or on Nordvik Region's citizen-service assistant in a separated test range with synthetic data. State who approves and on what date, what is in scope and out of scope, which catalogued benign scenarios may run, whether any AI component takes part and at what level, the stop conditions, and who holds the stop path and from where. Then, for three steps described only by technique name, write the expectation per layer before the run and the defensive improvement you would make if the expectation is not met.

Deliverable: A one-page manifest outline with approvals, scope, AI participation, stop conditions and stop holders, and three written expectations with their defensive follow-up.

Your work

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

In the continuous emulation pipeline template of AI Purple, at which level do scenarios above safety class S1 run?
Show the answer
D. Level 2 Supervised
Delegation is explicit in the template: safety class S1 inside scope at Level 3 Bounded, anything above S1 at Level 2 Supervised, and Level 4 never.
What limit does the chapter state for open test libraries and breach-and-attack simulation alike?
Show the answer
B. They show that a control reacted to a catalogued behaviour, not that anyone acted on it
The Summary states that the libraries and the product class give cadence and breadth, and that their limit is the same: a reaction to a catalogued behaviour, not proof that anyone acted or that other behaviour is covered.
What did the first AI red-team campaign against Nordvik Region's citizen-service assistant find under objective OBJ-03?
Show the answer
A. A page withdrawn from the public website was still in the retrieval index and was cited as current
The significant finding was that withdrawal of a page did not trigger re-indexing, so the assistant cited a withdrawn page as current. The finding ends in a defensive change to the indexing process.