A Reference Architecture for AI-Enabled Cyber Defence

All courses / AI Red Team / Module 1 / Chapter 4

Learning objectives

After the lesson the participant can:

• Compare the three placements of security AI and state what each loses.
• Explain why proposal, approval and execution of an action are kept in different components.
• Classify security data into the chapter's four classes and derive the permitted hosting for models.
• Design the placement of a new security AI capability, including range, identity, containment profile and untrusted inputs.

Key concepts

Security AI is AI

A triage assistant, a containment agent and an adversarial test harness all call models, read content they did not write and invoke tools. They run on the governed AI platform under the same controls as other AI systems, not beside it.

Proposed, approved, executed

An action is proposed in the agent layer, approved in the approval service outside the agent, by a person at Level 1 and Level 2 or by a codified playbook at Level 3 Bounded, and executed by the tool gateway, which re-checks the policy decision.

Classes of security data

Public intelligence, restricted intelligence, operational telemetry and sensitive case data, classified by content and not by the tool that produced it. The gateway reads the class and routes the request only to serving the policy permits; sensitive case data is self-hosted only.

Separated ranges

Production estate, blue tooling, red range and AI test range, each with its own network boundary, identities, credentials, tooling and evidence store. The red range has no standing route into production or blue tooling, and the AI test range runs copies, never production instances.

Findings are the only route

The one sanctioned route from the red range and the AI test range into blue tooling is the finding, a versioned record closed by a control and verified by re-running the scenario. Offensive tooling never crosses, but the detections derived from it do.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. Why does the chapter argue that security AI needs the platform's controls more, not less, than other AI systems?
2. The chapter says a capability that needs both a defensive and an offensive range is two capabilities. What practical difficulties does that separation create, and why accept them?
3. In what sense is keeping alerts on controlled capacity a defence and not only a compliance matter?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. What is lost when security AI is placed beside the platform, with the security team running its own model endpoints and direct API keys?

   1. Only the visibility of the vendor's models and data flows
   2. Nothing essential; this is the reference placement
   3. Gateway, policy point, evidence and containment
   4. The ability to self-host models in a national data centre
   Show the answer

   C. Gateway, policy point, evidence and containment
   The placement table says that beside the platform the gateway, policy point, evidence and containment are lost, and the security estate becomes the least governed AI in the organisation.

2. Under the default hosting in the chapter's class table, where may a model run on sensitive case data such as special-category data or employee investigations?

   1. On self-hosted serving only
   2. On self-hosted or EU-hosted serving under contract
   3. On any serving the analyst selects for the case
   4. On cloud serving if the request is pseudonymised
   Show the answer

   A. On self-hosted serving only
   The class table gives self-hosted only as the default for sensitive case data, and the gateway routes by classification whatever the analyst would prefer.

3. Why do Mode 3 Agentic work tools used by detection engineers and red-teamers run only in isolated ranges?

   1. Because coding agents cannot be connected to the AI gateway
   2. Because the AI Act classifies coding agents used in security work as high-risk AI systems
   3. Because isolated ranges have more reserved model capacity
   4. Because an agent with an engineer's credentials should not hold any that reach production
   Show the answer

   D. Because an agent with an engineer's credentials should not hold any that reach production
   The chapter states that an agent acting with an engineer's credentials should not hold credentials that reach production, so Mode 3 tools are confined to isolated ranges.