All courses / AI Red Team / Module 4 / Chapter 15
Learning objectives
After the lesson the participant can:
- Explain the three frictions AI has removed from social engineering and the three consequences for training.
- Design an exercise that tests an approval process, not individuals, within the chapter's four ethical principles.
- Select aggregate measures of resilience and justify why the click rate is not among them.
- Describe the skills analysts need to supervise security AI: verification, resisting automation bias, stopping agents and skill retention.
Key concepts
- Three frictions removed
- AI has not created a new class of social engineering; it has removed language, personalisation and presence as limits. The techniques are familiar; their cost and quality have changed, so recognition is no longer a reliable control.
- Test the process, not the person
- An exercise asks whether a payment-change process, a service-desk reset or an approval chain behaves as designed when a realistic AI-generated request arrives. People take part knowingly, and individual performance is not measured.
- Exercise design record
- A classified record with the process and control under test, hypothesis, persona, excluded themes, announcement window, consent, data handling, safety mechanisms and success criteria. It deliberately contains no call script and no synthetic media.
- Four aggregate measures
- Process adherence, report rate, time to report and time to containment, taken from systems that already record them. Results are compared over time for the same process, not between units.
- Unassisted practice
- To keep the skill the AI performs, by default ten per cent of alerts, chosen at random, are worked without the AI's proposal and compared afterwards, which trains the analyst and produces evaluation data for the AI.
Interactive exercises
Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.
Interactive exercise 1 · Match
Four measures of resilience
Match each aggregate measure to what it shows about resilience to deception.
Interactive exercise 2 · Scenario
A convenient proposal
Apply the chapter's verification habit as an analyst at Nordvik Region.
Teaching plan
| 10 min | The teacher presents the Arup case as a process failure and the sober reading of the threat reporting: AI added to existing playbooks, with three frictions removed. |
| 15 min | Discuss the four ethical principles, labour law and data protection for exercises, using Nordvik Region's design record NR-AWR-2027-01. |
| 15 min | Review analyst training: the verification habit, automation bias and approval quality from the override log, stop-path drills and skill retention. |
| 25 min | Exercise: groups write an exercise design record for one approval process, without scripts or media. |
| 10 min | Each group states what finding it would raise if the hypothesis failed, and whether a system change or training closes it. |
Discussion
- Does announcing an exercise at unit level remove its value? What does the chapter answer, and do you agree?
- Nordvik Region's first exercise had a report rate of 0.74 yet the hypothesis failed. What does that teach about where controls should be enforced?
- A rejection rate near zero on AI proposals is treated as a finding about the oversight design, not the analyst. Which design changes would you try first?
Written exercise
Exercise · 25 minutes
Write a deepfake-resilience exercise design record
Choose one high-impact approval flow in your organisation or Hansa Mutual's treasury approvals. Write its designed control: expected channel, call-back, second approver, cooling-off. Write the hypothesis with a reporting time. Describe the scenario only at the level of pretext and channel, with a fictitious or consenting persona and the excluded themes. Record the announcement audience and window, consent from employee representatives and the data protection officer's review, data sources, minimum group size, retention, whether calls are recorded, the payment hold and who can stop the exercise, and the success criteria. Do not write a call script or describe how synthetic media are produced.
Deliverable: A design record in the structure of NR-AWR-2027-01 with process, hypothesis, scenario limits, announcement, consent, data handling, safety and success criteria.
Self-test
Choose an answer to see whether it is right, or open the answer below each question.
Which of the following is among the chapter's four ethical principles for resilience exercises?
Show the answer
C. Announce the exercise at the level of the unit, without date or form
The principles are: announce at unit level, obtain specific consent from anyone whose likeness is used, exclude themes that exploit personal vulnerability, and debrief everyone. Calls are not recorded and results are aggregated.How was finding NR-FND-2027-014 from Nordvik Region's first exercise closed?
Show the answer
D. By a system change that enforces the call-back field
The payment system allowed a payee change to be saved before the call-back field was completed, so the finding is closed by a system change that enforces the field, not by a training module.Which kind of AI proposal are analysts trained to scrutinise in particular?
Show the answer
B. Proposals unusually convenient for an attacker, such as closing a case or allow-listing a sender
Because much of what security AI reads is attacker-controlled and can steer a proposal, analysts learn to scrutinise proposals that are unusually convenient for an attacker, such as closing a case, allow-listing a sender or taking no action.