Cover of AI Red Team

Book 4 · Lectures

AI Red Team

A keynote on the whole book and one lecture for each part, each with an abstract, a timed outline and what the audience takes away.

9 lectures7.2 hours in all

The keynoteThe slide deck

All courses / AI Red Team / Lectures

  1. Keynote: AI as an active part of cyber defence: mandate, bounded autonomy and proof
  2. Module 1: Foundations: from threat picture to a governed place for security AI
  3. Module 2: Govern: who decides what security AI may do, and who can stop it
  4. Module 3: Identify: know the estate, the paths through it and whether the controls hold
  5. Module 4: Protect: controls that hold however convincing the input
  6. Module 5: Detect: grounded signals, bounded agents and analysts in command
  7. Module 6: Respond: faster work, human command and one record for every clock
  8. Module 7: Recover: from an assumed plan to a proven capability
  9. Module 8: Verification and improvement: evidence that the controls still hold

Keynote on the whole book · 45 minutes

AI as an active part of cyber defence: mandate, bounded autonomy and proof

For security leaders, architects, practitioners and decision-makers in European enterprises and public authorities

Attackers already use AI to find weaknesses, write convincing lures and run parts of an intrusion faster than a person can, so defenders cannot treat AI only as something to protect. This lecture presents the argument of AI Red Team in one sitting. AI strengthens defence across every CSF Function when it acts inside a written mandate, at an autonomy level matched to the reversibility of each action, on a platform where containment holds whatever anyone intends, under continuous verification, and in a purple loop that turns every finding into a verified control. Two fictional organisations show what that means for a small public authority and a multinational insurer.

6 minThe defender's position has changed
  • Exploitation commonly precedes the patch, and AI carries out operational work in some intrusions under human direction.
  • AI does not fundamentally change the techniques attackers use; it reduces the time required to combine them.
  • Active defence means AI doing work people cannot do at the required speed or scale, inside boundaries people set.
6 minA complete map and four rules
  • NIST CSF 2.0 with three AI focus areas keeps attention on all six Functions, not only detection and response.
  • The book is defensive; every offensive activity is authorised and scoped; AI that acts is contained; every finding ends as a verified control.
  • European obligations are mapped once to CSF outcomes, so a control assessed once serves every regime.
8 minWho decides: usage modes and autonomy levels
  • The usage mode says whose identity the AI uses; the autonomy level says who decides each action.
  • Autonomy is set per action and earned by reversibility, bounded blast radius, measured confidence and a decision of the right tier.
  • Mode 0 is the trap: vendor features may already act at an effective autonomy nobody in the organisation chose.
8 minEverything security AI reads may be written by the adversary
  • Alert fields, email bodies, log lines and threat reports are channels for prompt injection, which is a property of the architecture.
  • The ceiling on consequences is set outside the model: policy decision point, permission manifest, task-scoped credentials and a stop path people hold.
  • The red team's own AI is a system to contain, and the organisation's AI is red teamed as a compound system.
7 minRecover: where AI changes the economics most
  • Recovery is neglected because testing it is expensive; AI makes dependency mapping, sequencing and rehearsal cheap enough to repeat.
  • Restore from known-good definitions and choose the restore point in relation to the incident.
  • Fast, recently proven recovery takes the leverage out of destructive and extortion attacks.
6 minProof: chaos experiments and the purple loop
  • Controls fail silently when the estate changes; chaos experiments inject the conditions controls exist for and check steady-state hypotheses.
  • A finding is finished when the control that closes it has been re-tested, not when it is reported.
  • The CSF profile as code moves only with evidence and gives the management body the numbers the teams use.
4 minTwo organisations and where to start
  • Nordvik Region applied every conclusion without a single Mode 5 agent; Hansa Mutual earned Level 3 Bounded for two reversible actions.
  • Start with the mandate, then the inventory of what already acts automatically, then the evidence.

What the audience takes away

Module 1 · Part I · Foundations · 45 minutes

Foundations: from threat picture to a governed place for security AI

The lecture carries the argument of Part I in four moves. Exploitation now commonly precedes the patch and AI carries out operational work in some intrusions, so defenders must put AI to work under control. The evidence is read with labels, observable, expected or speculation, so that budgets follow what was seen. NIST CSF 2.0, read through three AI focus areas, gives a complete map into which European obligations are mapped once. Finally, security AI is AI: it belongs on the governed platform, behind the gateway and the policy decision point, with separated ranges for offensive capabilities.

8 minWhy AI must become an active part of defence
  • Three documented developments: the speed of exploitation, AI moving from aid to operator for attackers, and AI finding and patching vulnerabilities for defenders.
  • An attacker needs one path that works; a defender must close many while the window narrows.
  • Active defence does not mean handing the security operation to a model: AI works inside boundaries people set, at an autonomy level matched to the consequences of a mistake, with evidence.
10 minReading the threat evidence without headlines
  • Three bands of ATT&CK tactics by strength of evidence: observed at scale, reported in AI-orchestrated campaigns, tooling seen with limited evidence.
  • AI does not fundamentally change the techniques; it reduces the time needed to combine them, and current frameworks trade stealth for velocity.
  • Mean time to exploit estimated at minus seven days: the controls that matter for the exposed estate work before a patch is installed.
  • Every threat-profile entry is labelled observable, expected or speculation, and the label moves only on evidence.
10 minCSF 2.0 as the map, European law as the obligation
  • Six Functions, 22 Categories and 106 Subcategories describe outcomes, not controls; CSF is a framework, not law.
  • The Cyber AI Profile adds three questions to every Category: securing AI systems, AI-enabled cyber defence and thwarting AI-enabled attacks.
  • Assure once, comply many: one control mapped to NIS 2, DORA, the Cyber Resilience Act and the AI Act, with evidence distinguished from attestation.
  • Tiers describe risk-management practice as a whole and are not a maturity score for each Category.
12 minSecurity AI is AI: the reference architecture
  • Three placements: beside the platform, inside the vendor tool, on the platform; only the last keeps gateway, policy point, containment and evidence.
  • Six layers from telemetry to actions, with proposal, approval and execution kept in different components.
  • Security data is classified by content and model location is a routing rule at the gateway.
  • Four separated ranges, with the finding as the only route from the red range and the AI test range into blue tooling.
5 minThe four rules and the two organisations
  • The book is defensive, every offensive activity is authorised and scoped, AI that acts is contained, and every finding ends as a verified control.
  • Nordvik Region stretches a team of fourteen and keeps security data on capacity it controls; Hansa Mutual seeks bounded autonomy at machine speed across seven markets.

What the audience takes away

Module 2 · Part II · Govern · 50 minutes

Govern: who decides what security AI may do, and who can stop it

The lecture follows one chain of authority through Part II. The management body approves a risk appetite for security AI in three dimensions, error, autonomy and reversibility, and sets a ceiling per usage mode. Within the ceiling, a named authority assigns a level to each action, and human command is built from a stop path, an override log and measured approval quality. Policy makes the chain enforceable, with rules of engagement as the signed gate for all offensive work. Oversight then shows the management body what is actually in use, including supplier AI whose effective autonomy nobody in the organisation chose.

10 minContext and appetite: write it down or delegate it to vendors
  • The context record names AI the mission depends on, security AI the defence depends on, their suppliers, and adversaries who use AI.
  • Appetite has three dimensions: error tolerance on the organisation's own evaluation set, autonomy ceilings per usage mode, and reversibility with blast radius.
  • A ceiling is not the level a system runs at but the limit within which a level may be raised on evidence.
  • Assumptions built for slower adversaries are revised explicitly: time to exploit, trust in voice and video, volume, deterministic tooling and adversary speed.
12 minModes, levels and the line between Level 2 and Level 3
  • Red, blue, purple and white describe functions, not departments, and AI changes the work of each but not its accountability.
  • The level is set per action: one agent can enrich at Level 4 while its containment proposals wait at Level 2 Supervised.
  • Five factors decide the level: reversibility, blast radius, confidence, exposure to attacker-controlled input and the decision tier.
  • A Level 3 capability without a codified playbook enforced outside the model is Level 4 with a misleading label.
8 minHuman command by design
  • The interruption path sits outside the system it stops and its holders are named, including an out-of-hours provider's shift lead.
  • The override log and approval-quality metrics are the evidence that command is real; approvals given in seconds signal rubber-stamping.
  • The response to rubber-stamping is redesign, not blame.
10 minPolicy and rules of engagement as code
  • Three layers: a statement the management body approves, standards per usage mode, and enforceable rules at the gateway, policy decision point and pipeline.
  • Each mode has its own instrument, from supplier assurance for Mode 0 to permission manifests and rules of engagement for Mode 5.
  • Offensive AI capabilities are dual-use controlled assets: registered, restricted, bound to a mandate, contained, logged and limited in autonomy.
  • The policy decision point checks every AI-proposed action against the signed manifest, and an AI action outside the plan is a stop condition.
10 minOversight and the AI supply chain
  • The management body sees an inventory view, risk indicators and decisions requested, generated from records.
  • Effective autonomy is the level at which a supplier feature actually acts; undeclared, it is a supply-chain finding in its own right.
  • Providers, vendors and MSSPs each test their own part; only the deployer tests the compound system in its own context.
  • Version telemetry, canary sets and verdict-distribution monitoring catch the changes suppliers do not announce.

What the audience takes away

Module 3 · Part III · Identify · 55 minutes

Identify: know the estate, the paths through it and whether the controls hold

Identify is where the book places red teaming, because CSF puts improvements from tests and exercises under this Function. The lecture follows the dependency chain of the part: no assessment without an inventory in which agents are recorded with identity, permissions and untrusted inputs; no ranking of paths without that inventory; no patch priority without exposure. It then turns to authorised emulation at the tempo of an AI-enabled adversary and to red teaming the compound AI system. The recurring principle is that AI proposes, deterministic components verify and act, and named people decide, with every finding closed by a re-run.

10 minThe inventory as a control
  • Seven linked asset classes: AI systems, models, agents, prompts and skills, connectors and tools, data, and security AI.
  • Reconciliation sorts every difference between declared and discovered into match, drift, orphan or unknown, each with an owner and a deadline.
  • The record becomes a control when the gateway generates an agent's tool allow-list from it.
  • Criticality follows reach, identity, exposure, data and dependency, which places most security AI at tier-0.
10 minContinuous threat modelling and reviewable machine-generated risk
  • Threat models are regenerated from architecture code on every change; AI drafts candidates and a named person accepts or rejects each with a reason.
  • Attack-path finding stays a deterministic graph computation; AI proposes edges, ranks paths with checkable explanations and proposes the cheapest cut.
  • Soundness is defined in advance in seven checks, and the completeness check catches the changed boundary without any candidate threat.
10 minFrom finding to fixing: the verification bottleneck
  • The constraint has moved from finding vulnerabilities to verifying, prioritising and fixing them.
  • No AI finding reaches a person until a machine has tried to reproduce it, and no fix is accepted until it is proven.
  • EPSS, the KEV catalogue and the EUVD say how dangerous a vulnerability is in general; exposure and business context say how dangerous it is here.
  • Delegated patch decisions for known-exploited, reversible fixes make patch velocity a service level.
13 minAdversary emulation under rules of engagement
  • A threat-informed scenario maps each step to an ATT&CK technique and names the detection or control it is expected to meet.
  • AI that executes emulation steps is treated as a Mode 5 system for the exercise, with exercise credentials, range separation, an external stop path and logging outside the red range.
  • The three controls that failed in July 2026 each have a counterpart in an emulation programme and a design response.
  • The periodic exercise is discovery; continuous validation is regression; they meet at closure.
12 minRed teaming the compound AI system
  • The model is the smallest part: most failures that matter arise where two parts of the compound system meet.
  • Taxonomy entries become objectives written for the control that should hold, and the adversarial inputs stay in the access-controlled scenario library.
  • For agents the permission manifest is the test plan, and a scripted driver tests the boundary without needing the model to misbehave.
  • A boundary does not know it is being tested, which is the reason to weight containment over behavioural testing.

What the audience takes away

Module 4 · Part IV · Protect · 50 minutes

Protect: controls that hold however convincing the input

The argument of Part IV is that protection can no longer depend on anyone recognising a forgery, whether the anyone is a finance clerk facing a synthetic voice or a model facing a crafted alert field. Each chapter therefore moves the control from recognition to structure: authentication that leaves nothing reusable to harvest, approvals that cannot be satisfied by the content of a request, guardrails that limit what any input can cause, and segments and degradation modes that bound reach and failure. The lecture shows the same design move four times and what each costs to operate and to verify.

12 minIdentity: three populations and no ambient authority
  • People, conventional machine identities and AI agents fail differently; an agent's actions vary with input that someone else largely controls.
  • Agent credentials are task-scoped, short-lived, issued by a broker outside the runtime and never shared; the agent has no ambient authority.
  • Phishing-resistant authenticators make deception yield nothing reusable, and a phishable fallback left enabled undoes the migration.
  • Appearance is not authentication: call-back to a registered contact, dual control and out-of-band confirmation, enforced as policy as code.
10 minPeople: test the process, not the person
  • AI removed three frictions from social engineering: language, personalisation and presence.
  • Exercises are announced at unit level, use consenting or fictitious personas, exclude themes that exploit personal vulnerability and are debriefed.
  • Resilience is measured by process adherence, report rate, time to report and time to containment, not by click rates.
  • Analysts are trained to verify against the evidence, resist automation bias, drill the stop path and keep their unassisted skill.
12 minData and platform: the ceiling is set outside the model
  • Security data reveals people, weaknesses and the defender's blind spots, and AI creates new copies of it in prompts, indexes, caches and logs.
  • Six guardrail layers: input handling, retrieval scope, tools, output checks, budgets and evidence.
  • Injection classifiers reduce noise; the ceiling is set by tools, identity and output checks, which hold whether or not a manipulation is recognised.
  • Configuration is policy as code, with automatic reversion only for control classes reviewed in advance.
11 minInfrastructure: bounded reach and safe degradation
  • No SOC can guarantee a response faster than an attacker's next step, so each step must cross a boundary where it can be refused, logged and detected.
  • Redeploy from known-good definitions, which for an AI system include model version, prompts, guardrails, manifest, retrieval sources and the evaluation suite.
  • Security AI fails as unavailable, degraded, untrusted or compromised; the quiet states are the dangerous ones.
  • Stepping down is automatic and triggered from outside the AI; stepping up needs a person and a passing evaluation suite.
5 minVerifying Protect
  • Each chapter ends with a chaos experiment that injects a condition, not an attack: an over-scoped token, a synthetic payee change, a canary record, a blocked model endpoint.
  • Nordvik Region's protection rests on contracts, process controls and fallback to its people; Hansa Mutual's on engineered credentials, policies and redundancy.

What the audience takes away

Module 5 · Part V · Detect · 45 minutes

Detect: grounded signals, bounded agents and analysts in command

The SOC is where most organisations first put AI to work and the one function that must not be misled, because much of what it reads is written by the adversary. The lecture argues that AI adds most value in reading and assembling and most risk in deciding and acting. It follows an alert from intake to closure, then goes upstream to the three disciplines that decide which alerts exist and how far they can be trusted. The common thread is that reasoning becomes an artefact a person can check: a grounded record, a rationale, a grade, a register entry.

8 minWhy the SOC needs AI and why it must be careful
  • MTTD, MTTR and the false-positive ratio are the yardstick: AI is justified when it moves them without degrading the quality of decisions.
  • Alert titles, email subjects, file names and command lines are fields an adversary can influence, so every component that reads alerts reads attacker-controlled data.
  • The workflow is taken apart into intake, triage, enrichment, investigation, escalation and closure, each with its own starting autonomy level.
10 minGrounded triage and the investigation agent
  • A verdict in prose is a liability; a grounded record cites evidence for every statement and lists what could not be established.
  • An investigation agent is a Mode 5 system governed as a well-meaning insider through a permission manifest enforced outside the model.
  • Five conditions for acting without per-action approval: reversible, deterministically high-confidence, bounded, a person informed with a stop path, and authority granted by the forum that owns the risk.
  • The model's own confidence is not a criterion.
8 minWatching the watchers
  • The gateway is the best sensor for the organisation's AI; what an agent says it did is a claim, what the gateway recorded is evidence.
  • No AI component in the SOC triages alerts about itself.
  • Approval latency, AI and analyst agreement, and AI-closed alerts later found to be true positives show whether the SOC became faster or only faster and worse.
10 minDetections and intelligence as code
  • A detection the organisation cannot read, review and test is a dependency, not a capability.
  • A detection has a rule, a technique mapping, test cases, a rationale naming the variants not caught, and control mappings.
  • AI drafts are plausible by construction; tuning is detection change; the assistant has no deployment rights.
  • An AI summary is not a source: people assign the grade, and a combined claim takes the weakest grade.
9 minDeception and its legal boundary
  • The value of deception lies in its false-positive rate: a touched honeytoken needs little weighing.
  • Decoys in AI systems catch both external attackers and the organisation's own agents exceeding their mandate.
  • Decoy alerts bypass triage to the role that can act, and reversible responses can be pre-approved at Level 3 Bounded.
  • No decoy, detection or playbook contains a step that acts outside the estate.

What the audience takes away

Module 6 · Part VI · Respond · 45 minutes

Respond: faster work, human command and one record for every clock

An incident tests every earlier investment under pressure, and AI can shorten almost every step while also producing a confident, plausible and wrong account of what happened. The lecture sets out how Part VI keeps the speed and removes the danger. Declaration runs against criteria written in advance; analysis is layered so that findings rest on re-runnable data and an analyst's signature; the incident record is the single source from which every regime's clock starts; and containment at machine speed exists only as a short catalogue of reversible actions that a policy point checks and a person can stop.

9 minDeclaring and coordinating at machine speed
  • Declaration criteria are agreed, versioned and tested beforehand, and prefer early declaration with a later downgrade.
  • An alert that reaches someone who cannot act on it has, in practice, not arrived: decision rights are explicit before any incident.
  • AI is the scribe's assistant, but people write or confirm the record of decisions that a supervisor or court will read.
9 minDefensible analysis
  • Every timeline entry points to the events it derives from, and the timeline shows its gaps because absence of evidence is a finding.
  • Chain of custody extends to the analysis: input data, instruction, model, output and human judgement.
  • A finding is reproducible when another analyst reaches the same conclusion from the recorded queries and data, not when the model repeats its words.
  • No finding about a person rests on AI alone.
9 minWhen AI is part of the incident
  • Four roles: target, channel, subject and instrument of analysis, recorded in an AI classification step at declaration.
  • An agent as subject is stopped through the stop path, then preserved; evidence comes from outside the agent; the subject does not investigate itself.
  • Impact queries are versioned with the system and read the log written by the policy decision point, not the agent's own account.
9 minFour regimes, one record
  • NIS 2 starts from awareness, DORA from classification as major, the AI Act from a causal link or its reasonable likelihood, and the GDPR from awareness of a personal-data breach.
  • A clock starts on the earliest plausible trigger and stops only by recorded decision, never by silence.
  • AI drafts from analyst-confirmed facts and has no submission rights; a named signatory approves every external statement.
  • Reports under different regimes are reconciled so that differences are explained by new facts, not different wording.
9 minContainment and eradication
  • A catalogue lists each containment action with autonomy level, scope limits, reversal and evidence.
  • The containing agent is itself a privileged insider: own identity, policy outside the model, budgets and a stop path held by a named person on every shift.
  • Eradication rebuilds from known-good definitions, remains a human decision and is complete only when verified.

What the audience takes away

Module 7 · Part VII · Recover · 50 minutes

Recover: from an assumed plan to a proven capability

Recovery is invisible until needed, owned by no single function and expensive to test, so most organisations learn how well they recover on the worst day of the year. The lecture argues that AI changes the economics of recovery without changing who decides. It moves through three steps: executing a plan held as code, with restoration from known-good definitions and restore points chosen in relation to the incident; restoring AI systems, where running again is not the same as behaving as before; and proactive recovery, in which drills measure recoverability and destructive and extortion attacks lose the leverage that the victim's doubt gives them.

8 minThe neglected Function and what AI changes
  • Three structural disadvantages: recovery is invisible, unowned and costly to test.
  • AI makes dependency mapping, sequencing and comparison with definitions cheap enough to run routinely.
  • The judgements stay with people: which service matters most this week, whether the adversary is gone and when recovery is over.
12 minExecuting a recovery plan as code
  • A plan states objectives, business impact tier, dependencies, restore sources, and verification and end criteria for each service.
  • The restore point is chosen before the earliest indicator of compromise, not as the latest backup.
  • Reversible steps run at Level 2 Supervised in production and Level 3 Bounded in rehearsal; irreversible steps stay at Level 1 Assisted.
  • Return to service needs two confirmations: the adversary removed, and the service fit to resume.
8 minRecovering in a contested environment
  • A cyber incident is not a passive disruption: an attacker with access can watch the restore and reinfect.
  • Restore the identity plane first, communicate out of band, reconnect in stages, protect the recovery tooling as tier-0 and watch for re-entry with decoys.
  • The clean room keeps restored systems apart from the production network and identity plane until they are verified.
12 minRecovering AI systems and security AI
  • The evidence bundle of the last approved release is the bill of materials for recovery.
  • Two components cannot be restored: agent memory has no known-good source, and credentials are replaced, never restored.
  • Retrieval stores are rebuilt from authoritative sources; golden datasets and the red-team suite, including the incident scenario, run before return.
  • Losing security AI must reduce speed but not capability: degraded modes and manual playbooks are kept current and exercised.
10 minProactive recovery
  • Recoverability is the measured ability to fulfil the plan, and every measurement has an age.
  • A prediction is never evidence of recoverability: it triggers a drill or a finding.
  • Drill findings go to Govern, Identify and Protect as pull requests and close only when the same drill passes on re-run.
  • Recovery communication rests on a message library, an approver rota, independent channels and release targets.

What the audience takes away

Module 8 · Part VIII · Verification, Improvement and Conclusions · 50 minutes

Verification and improvement: evidence that the controls still hold

Controls rarely fail loudly; they fail silently when the estate changes, and the organisation learns of it when an adversary exploits the failure. The last part of the book answers with two disciplines. Security chaos engineering verifies capabilities in every CSF Function by injecting conditions and checking steady-state hypotheses, on a platform whose safety boundaries are its design. The purple loop turns every finding, from whatever source, into a control that has passed a re-test. The lecture then uses the two organisations and the eight conclusions to show that the same programme shape holds at very different scales.

12 minSecurity chaos engineering
  • Red teaming is discovery, breach-and-attack simulation is regression for Protect and Detect, and security chaos engineering is continuous verification across every Function.
  • A steady-state hypothesis is falsifiable and measured from outside the control: condition, expected outcome, observation and time bound.
  • Most experiments contain no attacker technique; absence is a result, and the expected outcome is often an action by people.
  • AI plans, varies and evaluates, but the AI that plans must never hold the credentials to execute.
8 minThe platform that must not become the incident
  • Three zones: planning with AI and no estate credentials, control with deterministic policy, scheduler, abort controller and ledger, and execution with narrow actuators.
  • Catalogued experiments only, reversible by construction, bounded blast radius, synthetic data, and Level 4 Autonomous never.
  • A sealed ledger keeps the SOC's trust: the shift lead can always ask whether an alert is an experiment.
12 minThe purple loop and what to measure
  • Five states: finding, control, detection, re-test and evidence, with one strict exit condition.
  • A deployed but untested control leaves the finding implemented, unverified and reported as open.
  • Only verified coverage counts: a detection that fired in a re-test or chaos experiment in the last 90 days.
  • A component that saves time but raises AI-closed alerts later found to be true positives has made the SOC faster and worse.
8 minTwo organisations, one programme shape
  • Nordvik Region built the mandate before the tools, stayed at Mode 4 and below, and found its largest gains in Recover.
  • Hansa Mutual built control around the AI it already had and earned Level 3 Bounded for two reversible actions.
  • Both found the same blind spot: vendor AI already acting automatically without anyone having decided it.
  • Three tests for any organisation: the mandate test, the inventory test and the evidence test.
10 minEight conclusions and what remains open
  • The usage mode, not the model, sets the risk; autonomy is earned action by action, according to reversibility.
  • Everything security AI reads may be written by the adversary, so containment must not depend on intent, for defensive AI too.
  • Three tensions remain: speed versus control, autonomy versus accountability, innovation versus sovereignty.
  • Open: whether evaluation can carry assurance when models recognise test settings, and how far human verification keeps pace.

What the audience takes away