Technology Infrastructure Resilience Against Machine-Speed Attacks

All courses / AI Red Team / Module 4 / Chapter 17

Learning objectives

After the lesson the participant can:

Explain why the infrastructure must limit the reach of each attacker step and list the properties that do so.
List what a known-good definition contains for an AI system and why a redeployed system starts at reduced autonomy.
Distinguish the four ways security AI fails and identify the two quiet ones.
Design degradation modes for a security AI capability with entry triggers, fallback, staffing and fail behaviour.

Key concepts

Bounded reach first: No SOC can guarantee a response faster than an attacker's next step, so the infrastructure limits the reach of each step: the attacker needs many steps, and each crosses a boundary where it can be refused, logged and detected. Speed of response is the second line.
Redeploy, do not repair: Repairing a touched component means trusting that every change has been found; replacing it from a signed definition built by a pipeline means trusting the definition, a far smaller and more verifiable claim. Regular rebuilds also limit persistence.
Four failure states of security AI: Unavailable, degraded, untrusted, and compromised or out of mandate. Degraded and untrusted are dangerous because they are quiet: the system keeps producing plausible output, and untrusted is the state an attacker will try to induce.
Lose speed, never the defence: The design principle with three rules: fall back to people and deterministic playbooks, reduce autonomy in steps, and trigger degradation from outside the AI. Stepping down is automatic; stepping up requires a person's approval and a passing evaluation suite.
Fail safe, not fail open: The degradation-mode definition records for each function whether it fails open or closed and who accepted that choice. The fallback includes deterministic controls that catch the worst cases without the AI.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Interactive exercise 1 · Put in order

Stepping down from Level 3 Bounded

Put the degradation path of a security AI capability at Level 3 Bounded in order, from normal to last resort.

Interactive exercise 2 · Sort

Four ways security AI fails

Sort each signal or condition under the failure state the chapter would assign to it.

Teaching plan

15 min	The teacher presents why speed changes the design, the zero trust baseline and the table of properties that limit machine-speed attacks.
10 min	Participants list the contents of a known-good definition for one of their AI systems and mark which items are not yet versioned.
15 min	Walk through the four failure states and the degradation modes of HM-SEC-001, from normal through bounded-suspended and advisory to manual.
25 min	Exercise: groups define degradation modes for a security AI capability following the eight steps of the practical guide.
10 min	Groups pick one mode transition and write the chaos experiment that verifies it, with the fallback staffed in advance.

Discussion

Why is a rising rate of analyst rejections treated as a trust signal and not as a nuisance to be tuned away?
A package mirror was also an egress path in July 2026. How would you enumerate reachable paths from inside a segment instead of reading the intended rules?
Where does your security AI share accelerators, gateway or identity provider with production AI, and what does that coupling imply for your fallback?

Written exercise

Exercise · 25 minutes

Define degradation modes for a security AI capability

For a capability in your organisation or Nordvik Region's SOC triage assistant NR-SEC-001, list its functions and the autonomy level of each. For each function write the fallback that works without the AI: who performs it, with which playbook and with what added staffing. Define the modes as a sequence from normal to manual. For each mode give entry triggers as measurable signals from outside the AI with thresholds and windows, starting from the chapter's defaults. State that stepping down is automatic and who approves stepping up after a passing evaluation run. Record for each function whether it fails open or closed and who approves a fail-open choice. Name where the monitor runs.

Deliverable: A degradation-mode definition in the structure of HM-SEC-001-degradation with modes, triggers, fallback, staffing, fail behaviour and approver.

Your work

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

Which item of a known-good definition for an AI system does the chapter say matters most?
Show the answer
B. The evaluation suite and thresholds it must pass before returning to its normal autonomy level
A system redeployed from correct definitions can still behave differently if a hosted model changed or an index was rebuilt from poisoned content, so it starts at reduced autonomy and returns to normal only after its evaluation suite passes.
In the degradation modes of HM-SEC-001, who or what decides the agent's mode?
Show the answer
A. A monitor on the SOAR platform, outside the agent's runtime
The monitor that evaluates the triggers runs on the SOAR platform, independent of the agent; the chapter states that the agent never decides its own mode, in line with detection from telemetry and not self-report.
By the chapter's default values, when does a capability enter advisory mode?
Show the answer
C. When analysts reject more than 30 per cent of proposals over four hours
The defaults are manual mode after 15 minutes of unavailability, suspension of bounded actions on an unannounced model change or evaluation regression, and advisory mode when analysts reject more than 30 per cent of proposals over four hours.