The green dashboard problem: verifying defence when AI sits on both sides
For security leaders, practitioners on the red and blue sides, architects, auditors and supervisors
An emulation side that can vary and repeat adversary behaviour, and a detection side whose triage and drafting are done by agents, can close a loop between themselves that only agrees with itself. The keynote sets out the book's answer: purple is a loop of five stages held by named roles, a finding is closed only on a passed re-test with a variant, coverage is a claim with an age, and a check counts only where tester and tested can fail in different ways. Two fictional organisations, one that buys the loop and one that runs it, show what the design looks like after a year.
6 min
Two nights, one technique
The same discovery technique, run by a contracted tester at Nordvik Region and by a pipeline at Hansa Mutual.
Novelty with a dated, narrow result against repetition at the pace of change.
The question for both: what can be proved later, and who checked it.
7 min
Why the annual rhythm no longer holds
Six testing forms, each leaving a stage unfinished; the report was the end of the loop in practice.
A shrinking exploitation window, an estate that changes fortnightly or daily, and detections that decay between tests.
AI makes repetition, variation and drafting cheaper, not novelty, judgement, independence or safety.
8 min
The loop and its closure rule
P1 Target, P2 Emulate, P3 Observe, P4 Improve and P5 Verify, each defined by a record.
Closed means a passed re-test with a variant, attached evidence, a named signer and a date.
Emulation runs inside a signed manifest enforced outside the agent, with a stop held by named people.
9 min
Independence
Shared model family, data, authorship, vendor or time makes errors correlate and coverage look better than it is.
Held-out scenarios, seeded gaps and reproduction by a person show that a check can fail.
Hansa Mutual's planner and drafter shared a family; a held-out re-test moved two verified cells to failed.
Machines alone can show that the known still works, not that coverage is real.
8 min
Coverage as a claim, and the people who make it
Report the denominator, the reasons and the age: two of nine at Nordvik Region, 27 of 34 at Hansa Mutual.
Measures that running more scenarios cannot improve, and never a measure of a named person.
A protected share of human-led work and a designed route to the red craft.
7 min
The verdict and Monday
No part of the thesis earned a plain held; six of twelve questions stay open.
A bought loop is worth the terms on which its evidence can be inspected; a run loop is as good as its independence and its stop.
One step for Monday: ask of one closed finding its re-test date, its variants and its signer.
What the audience takes away
Purple is the joint loop between the red team and the blue team, and its unit of work is the verified finding.
Coverage is a claim with a technique, an environment, a result, a date and an expiry; untested is not covered.
A check is evidence only where the side that tests and the side that is tested can fail in different ways, and a person can reproduce the result.
Module 1 · Part I · The Loop in Question · 45 minutes
From the report to the loop: why purple is not a team
The lecture carries the argument of Part I in one line: a test that ends in a report proves less than its name suggests, and the pressure on that rhythm is now measurable. It starts with two Thursday-night runs of one technique at Nordvik Region and Hansa Mutual, shows where the six traditional testing forms stopped, sets the shortening exploitation window and detection decay against the annual calendar, and ends with the loop the book defines: five stages named by their records, a closure rule with five states, ceilings on delegation and a principle of independence for the case where AI is on both sides of a check.
8 min
Two nights, one technique
T1087.002 run by a contracted tester in Nordvik Region's test tenant and by an emulation runner from Hansa Mutual's library of 86 scenarios.
The contracted run supplies novelty and a dated, narrow result; the pipeline supplies repetition at the pace of change.
What matters in both is what the organisation can later prove, and who checked it.
9 min
Where the old forms stopped
Six forms, each answering a different question, each leaving one stage unfinished in the common case.
The report was the end of the loop in practice although the guidance had long advised retesting.
Findings stalled because closure had no owner and no test; this is the book's own conclusion.
9 min
The pressure, measured
Time-to-exploit reported as 63 days, then 32, then 5, against a test bought once a year.
The test-to-change ratio: about 26 for Nordvik Region's benefits-triage service between two annual tests.
Detection decay reported in three states, current, at-risk and expired, and never as one.
11 min
The loop defined
P1 Target, P2 Emulate, P3 Observe, P4 Improve and P5 Verify, each defined by a record and not by a meeting.
Closed and verified is the only closed state; implemented but unverified and risk accepted are reported separately.
Ceilings per stage and action class, with Level 4 Autonomous never used for the four excluded classes.
Closure is always a named person's signature; the runner proposes and never closes.
8 min
Independence and what the loop is not
A check counts only if the side that tests and the side that is tested can fail in different ways.
Five kinds of separation, one incentive rule between roles and three proofs.
The loop is not a team, not an event and not a dashboard.
What the audience takes away
The unit of work is the verified finding, not the report, the session or the passing run.
An annual rhythm cannot follow an estate that changes fortnightly or daily; the test-to-change ratio shows by how much.
Purple is the joint loop between the red team and the blue team, held by named roles, and never a third team.
Module 2 · Part II · The Stages of the Loop · 60 minutes
One finding through five stages: the craft of the purple loop
The lecture follows a single finding around the loop to show why each stage has the record it has. It begins with the choice of what to test and the hazard of a library that grows towards what is easy to emulate, moves through the signed manifest and the stop path that make emulation safe, the campaign and the pipeline, the sealed expectation and the seven outcomes, the routing of a finding to the right change, and the re-test with variants that decides closure. Nordvik Region's spray-then-success finding, detected by the Region and closed by its provider's AI, runs through the whole hour.
8 min
P1 Target: choosing by risk, not by ease
Four inputs: intelligence, the threat register, risk assessment and the organisation's own incidents and findings.
Priority techniques ranked by impact, relevance and uncertainty about the defence, in coarse tiers.
The party that proposes scenarios is not the party that selects them; a curator proposes at Level 1 Assisted.
10 min
Rules before scenarios
Three documents kept apart: rules of engagement, mandate manifest and decision record; the lowest ceiling governs.
Scope as code: default deny, exclusions win, the manifest outside the agent's reach, refusal tested by three dry runs.
Safety classes S1 to S4 decide what may run where and at which level.
The stop path denies, revokes and cleans up, and the emulating system takes no part in it.
10 min
P2 Emulate: the campaign and the pipeline
The emulation plan binds scenarios to the signed scope and gives the blue team ground truth without the plan.
People contribute judgement that depends on consequence and context; AI adds breadth, speed and lower cost.
A pipeline is exposed to Goodhart's law; the held-out share and the generalisation gap are its honest number.
10 min
P3 Observe: expectation first
The expectation sheet is written from the detection catalogue and sealed before the result is shown.
Seven outcomes ordered by layer; detected demands the whole chain through triage and action.
Nordvik Region's step 2: the rule fired at 02:17 and the provider's AI closed the alert at 02:21.
10 min
P4 Improve: the right change, owned and reviewed
Routes by the layer that failed: detection, control, playbook, mandate or recorded risk acceptance.
The tester who authored a scenario does not tune the detection that answers it.
Review is the bottleneck by design; cost is counted by replay, shadow and an alert budget.
12 min
P5 Verify: closure with an age
The exact re-test is necessary and insufficient; variants, a held-out variant and a negative control test the technique.
Nordvik Region's exact re-test passed in November 2027 and an out-of-hours variant failed in January 2028.
Evidence is generated, hashed, versioned, signed and minimised, and it expires by clock and by condition.
Report the denominator, the reasons and the age: 25 of 34, not 74 per cent.
What the audience takes away
Each stage is defined by a record that the next stage can use without the meeting that produced it.
Safety and independence are decided in the manifest, the plan and the regression case, not left to the care of an operator or an agent.
A merged change is not a closed finding; closure needs a passed re-test with a variant, attached evidence, a named signer and a date.
Module 3 · Part III · The Loop Meets AI Systems · 40 minutes
When the target and the defender are both AI systems
The lecture argues that an AI system is tested and watched as a compound system, model, tools, data and identity together, and that a passing evaluation is not a running detection. It then turns to the blue team's own AI and makes the part's second claim: a delegated level on the defending side is a standing claim that has to be earned repeatedly, against cases whose answer the agent's operators did not write, and lost quickly on failure. Nordvik Region's withdrawn page in the citizen-service index and Hansa Mutual's benign twin that lowered an agent's level carry the argument.
8 min
A second question for every AI red-teaming test
A red team asks what the system can be made to do; the loop also asks whether the defence would have noticed.
Four classes of target by ATLAS technique: the model and its instructions, retrieval and the index, tools and actions, data and outputs.
Prevention and detection are recorded separately; a control that held but was not seen is a finding.
9 min
Telemetry and detection for agents
Six sources, from the AI gateway to the identity provider, each with its blind spot; the agent's narrative is not telemetry.
Detection is built from the manifest: a denied call and an allowed call outside the ceiling are separate conditions.
Detections cover an agent acting beyond its mandate, not an agent steered within it.
7 min
Evaluation is not detection
An evaluation speaks about a version under chosen inputs; a detection about production events nobody chose.
Each decays differently and each coverage claim carries a date and an expiry.
Nordvik Region's DET-NR-0011: a document in the index that the publishing register does not list.
9 min
Testing the defenders' AI
Six subjects, four properties: correctness, grounding, restraint and robustness to hostile input, with restraint first where action is delegated.
Emulation supplies seeds with a ground truth the agent's operators did not write, including benign twins.
The level falls at once on failure and rises only by a named authority's decision after a passing re-test.
7 min
The provider's AI and the limits of testing
An injected-test clause makes the Mode 0 blind spot testable without negotiating each test.
Nordvik Region's first round: 14 of 15 seeds correct, one escape, and a stop that took 21 minutes against 15.
A small sample can show an escape; it cannot show a small escape rate.
What the audience takes away
Watch an AI system as a compound system and state coverage per technique, per system and with an expiry.
A delegated level on the blue side is re-shown after every change of mandate, model or connector, with seeds authored by someone independent.
What a buyer cannot inspect it can still test, if the contract carries an injected-test clause, evidence duties and a stop right.
Module 4 · Part IV · The Loop's Own AI · 50 minutes
Can machines close the loop? Mandates, independence and the tier-0 set
The lecture carries the central design argument of the book. A team of agents can run much of the loop, but only if each agent is narrow, bounded by a mandate that a policy decision point enforces, and stoppable from outside. Even then, a chain in which the scenario, the detection and the verdict share a model family, data, an author, a vendor or a moment in time may only agree with itself. The lecture shows how independence is declared, tested and dated, why a person must be able to reproduce a result, and why the records the loop produces are an adversary's best map of the estate.
10 min
A team of narrow agents
Eight candidate functions; the runner executes a catalogue and does not compose, the triager ranks and does not close, the critic flags and never gates.
Four questions set every ceiling: reversibility, scope, personal data or a named person, and independence of the check.
Only the runner and the re-test runner reach Level 3 Bounded, and only for catalogued S1 scenarios.
8 min
The mandate as code
Identity of its own, tokens issued outside the agent, a catalogue binding, budgets and circuit breakers.
A never list closes the routes by which an agent could change its own governance; a separation list implements independence.
Authorisation, the stop, closure and the authorship of novelty are not delegated at any level.
12 min
Correlated failure and the five kinds of independence
What the research measured, and the inference to the loop that is the book's own.
Model, data, authorship, vendor and time: each with a shared dependency, a control and a piece of evidence.
A kind that is not shown is treated as shared; leakage of scenarios into detections needs no model at all.
Module 5 · Part V · The Loop in the Organisation · 60 minutes
Covered, measured, decided: the loop as an organisational claim
The lecture argues that the organisation's hardest questions about the loop are questions about claims and rights, not tools. Covered is a claim with five parts and a status computed by rule. Measures are safe only when running more scenarios cannot improve them. Whoever runs the stages, the coordinator, the closure owner, the white team, the register of record and the right to stop remain with the organisation. Ten decisions need named holders and times, the law turns into fields in a manifest and clauses in a contract, and every boundary the loop crosses needs consent, notice and a stop on both sides.
10 min
Covered is a claim with an age
Five parts: a technique, an environment, a result at a named layer, a date and an expiry.
Six statuses computed by rule; only verified counts as covered and a person declares.
Hansa Mutual's conversion: four cells that were green on the heat map had evidence past its expiry.
10 min
Measures that activity cannot improve
Six measures against four that mislead: scenarios run, findings opened, findings closed and a green percentage.
Goodhart's law in four mechanisms, and why AI makes the adversarial form faster.
Measures of the system, never of a named person; the catalogue is agreed with representatives before collection.
10 min
Who runs the loop and what must stay
In-house, contracted, hybrid and sovereign or sector models, chosen per stage with exit criteria.
Seven chartered roles; an AI system does not hold a role.
Nobody marks their own work; a small team compensates with countersignature, sampling and a contract.
10 min
Ten decisions and the white team
D1 to D10 with a holder, a deputy, a time, an evidence standard and a record.
Stop needs no approval and runs outside the system stopped; restart is a separate decision at a lower level.
No alert is closed as an exercise without a match in the tag ledger.
10 min
Regulation as a design constraint
DORA Articles 24 to 27, NIS 2 Articles 20, 21 and 23, the AI Act, the Cyber Resilience Act, the GDPR and co-determination.
Continuous emulation is not a threat-led test, and a hold list protects the test's secrecy.
The owner's written authorisation separates an exercise from an offence.
10 min
Across boundaries
Through a provider's pipeline, through its response chain, and never against its platform.
Share the technique and the expected evidence, never the target mapping.
Supervisors receive the records in a fitting form; the gap list is inspected and not copied.
What the audience takes away
Report the denominator, the reasons and the age; retire the heat map for reporting.
Buy capacity and a different view, and keep the coordinator, the closure owner, the white team, the register and the stop.
Write authority, consent, legal basis and notice into the manifest, and test the decision rights before the night they are needed.
Module 6 · Part VI · People and Career · 45 minutes
Two professions, one loop: incentives, craft and the next ten years
The lecture argues that the loop's records and mandates work only if the people who run it can say what they saw, keep the skills the automation no longer exercises, and know where independence will live as the organisation changes. It starts from the scoreboard of red wins when blue loses, sets out what a blameless session requires, states honestly how thin and adjacent the evidence on wellbeing and on skill beside automation is, and ends with three futures for 2036 that differ on where independence lives and agree that it is needed.
10 min
What red and blue are rewarded for
A scoreboard describes incentives, not characters; AI makes runs and drafts cheap, so measures that count them need a gaming check.
Accountability stays separate where independence needs it; the target is shared where learning needs it.
The shared target: priority techniques verified with evidence of a stated age.
9 min
Being tested, and staff exposed to exercises
A charter written before the session; roles and systems recorded, not names; anyone may pause.
Say what class of test exists and withhold only the instance; count the test against the system.
Staff are protected by governance and not by consent: representatives first, no individual consequences, a debrief the same day.
10 min
Keeping the red craft
The craft is the choice of path and the adaptation when a step fails; routine is running a known technique in a known form.
The evidence on skill beside automation comes from other fields, and none of it concerns red teaming.
Attempt first, manual-form practice on the range, rotation in both directions and a craft register tied to the runner's level.
9 min
Three futures for 2036
Two teams with one loop, one engineering practice, and a pipeline with a human verifier.
They differ on where independence lives, not on whether it is needed.
The verdict: that independence is needed is expected; how it is provided is speculation.
7 min
A register that can be wrong in a reviewable way
Each trend has a label, a stage, evidence, a trigger, a posture and an owner.
Triggers are computed from the loop's own records, such as the held-out catch rate and the age of evidence.
The author's projection is given with five thresholds that would change it.
What the audience takes away
Measure red and blue together on verified coverage with evidence age, and never turn a measure of testing or of load into a measure of a person.
The route to the red craft is a design decision that can be made explicit, funded and counted.
Decide where independence lives, write it down, and tie the view of the future to triggers and review dates.
Module 7 · Part VII · Experience and Conclusions · 45 minutes
What a year in the loop proves, and what it does not
The closing lecture asks what the book has earned the right to claim. It shows how loops fail where a step was reasonable and a question was not asked, tells the year in which every failure at Nordvik Region and Hansa Mutual was found by the loop and not by an incident, and gives the two answers of April 2028: two of nine techniques verified, with reasons, and a pipeline that does not make the session redundant. It ends with the author's verdict, in which no part of the thesis earned a plain held, and with a first step for each role.
10 min
Eight ways a loop fails
Four public cases: an authorised test that ended in arrests, stolen red-team tooling, evaluation agents beyond scope, and coverage figures from configuration.
Eight anti-patterns, each with the stage it breaks and the control that changes the outcome.
None of the cases shows that AI made the loop unnecessary or unsafe in itself.
10 min
A year with a failure at each organisation
Nordvik Region: an hours variant fails on 19 January and all variants pass on 9 February.
Hansa Mutual: a restraint failure on 24 January and an independence re-test on 22 February that moves two cells to failed.
Every recovery had three parts: contain at once, fix by one person, re-test by another.
9 min
Two answers and two plans
AIDR-0841: a verified share with its list, reasons and age, not closed findings.
AIDR-0891: keep the quarterly session for held-out and human-led scenarios.
A bought loop is worth the terms on which its evidence can be inspected; a run loop is as good as its independence and its stop.
9 min
The verdict on the thesis
Parts 1, 2, 4 and 5 held with qualifications; part 3 held in part and was restated.
The refutation conditions did not occur, but the cases are fictional and illustrate without testing.
Two operating rules carry the book: the closure rule and the independence principle.
7 min
Monday
One first step per role that takes a day or less and uses a record the reader already has.
The first-week plan: inventory, target, register, claim and rules, owners.
Six of the twelve questions stay open, and the author's projections carry triggers that would change them.
What the audience takes away
A loop that wants to be believed keeps records that someone who was not there can check.
Report what was tested with its age, and the list of what was not; neither organisation gave its board a green dashboard.
Keep the two rules: close only on a passed re-test with a variant, signed and dated, and count a check only where tester and tested can fail differently.
