Red Teaming AI Systems and Agents · AI Red Team: the course

All courses / AI Red Team / Module 3 / Chapter 13

Learning objectives

After the lesson the participant can:

Scope a compound AI system in six elements and explain why testing the model in isolation says little.
Write taxonomy entries as objectives for the control that should hold, with ATLAS technique and risk-register entry.
Explain how a scenario library, harness, threshold and gate turn a campaign finding into a regression case.
Design a containment test plan for an agent, with hypotheses for credentials, control artefacts, detection and the stop path.

Key concepts

Compound system: Users and content sources, a gateway, prompts and the model, a retrieval store, an agent with a permission manifest, and tools. Most failures that matter arise where two parts meet, and the content sources are the element most often forgotten in scope.
Objective for the control that should hold: Each applicable taxonomy entry is written as a statement such as: a resident cannot cause the assistant to disclose another resident's correspondence. The plan is a defensive artefact; the adversarial inputs live only in the access-controlled scenario library.
Threshold as a rate: Because generative systems are not deterministic, each case runs several times. The threshold is zero for outcomes that must never occur, and a non-zero rate only where other controls bound the harm, recorded with its owner.
Escape path and scripted driver: An escape path is any route by which an agent's effect can reach beyond its boundary. In the test range a scripted driver replaces the agent's planning component and issues out-of-mandate tool calls with the agent's own identity, testing the boundary reproducibly.
Evaluation awareness: Models may behave differently when a setting looks like a test, so a passing result may overstate production behaviour. The chapter prefers tests whose outcome does not depend on the model's behaviour, because a boundary does not know it is being tested.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Interactive exercise 1 · Put in order

An escape-path assessment

Put the steps of the chapter's escape-path assessment on a production agent in order.

Interactive exercise 2 · Match

Vocabularies and machinery of AI red teaming

Match each source or component to the purpose the chapter gives it.

Teaching plan

15 min	The teacher presents the compound system, the split between the provider's Article 55 evidence and the deployer's test, and the six scope elements.
15 min	Participants turn three taxonomy entries into control objectives for Nordvik Region's citizen-service assistant and mark one as not applicable with a reason.
15 min	Walk through library, harness, threshold and gate using GATE-CSA-001, then the withdrawn-page finding and its closing control.
35 min	Exercise: groups write a containment test plan for a production agent, modelled on the plan for Hansa Mutual's IT-operations agent.
10 min	Groups draft the residual-risk statement their system owner would sign and say what telemetry would let a responder reconstruct an out-of-mandate action.

Discussion

Instructions lowered the rate of harmful actions in the cited study without eliminating it. What follows for how much weight a passing behavioural test should carry?
Where in your organisation does the person who controls the content differ from the person who uses the assistant, and has that route ever been tested?
A CISO override of the release gate is valid for at most 14 days at Nordvik. What should have to be true for an override to be used at all?

Written exercise

Exercise · 35 minutes

Write a containment test plan for an agent

Choose a production agent in your organisation, or Hansa Mutual's claims-triage agent, and assume testing happens only on a replica in an AI test range under approved rules of engagement. From its permission manifest derive at least five objectives as hypotheses, covering credentials, tool scope, control artefacts the agent must not write, detection of decoy use, and the stop path while a task runs, each with a time bound where one applies. Enumerate three possible escape paths from the infrastructure description. State the two rounds, scripted driver and model with manipulated input, the actions never attempted, the telemetry that proves each result, and who owns the residual risk. Do not write any test inputs.

Deliverable: A containment test plan with five or more hypotheses, three enumerated escape paths, the two rounds, never-attempted actions, telemetry per objective and the residual-risk owner.

Your work

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

What was the significant finding of Nordvik Region's first campaign against the citizen-service assistant?
Show the answer
D. A page withdrawn from the public website was still in the retrieval index and cited as current
Under objective OBJ-03, a withdrawn page remained retrievable because withdrawal did not trigger re-indexing. The closing control made withdrawal remove the document from the index within one hour, verified by a new suite case.
Why does the escape-path assessment use a scripted driver in place of the agent's planning component?
Show the answer
A. To test the boundary reproducibly without needing the model to misbehave
The scripted driver issues out-of-mandate tool calls with the agent's own identity and runtime, so the boundary is tested reproducibly; a second round uses the real model with manipulated inputs to check detection and the stop path.
How does the chapter treat a threshold above zero in a gating suite?
Show the answer
C. It is allowed only where other controls bound the harm, recorded with its owner
Zero applies to outcomes that must never occur, such as one resident's data in another's answer. A non-zero rate is accepted only where other controls bound the harm, as with two per cent of unsupported answers that are labelled.