All courses / AI Red Team / Module 4 / Chapter 16
Learning objectives
After the lesson the participant can:
- Classify security data into the chapter's six classes and state the default handling of each when AI processes it.
- Explain why guardrails for security AI limit what any input can cause and do not rely on recognising malicious instructions.
- Draft a guardrail policy with the six layers for a security AI capability.
- Decide per control class whether configuration drift reverts automatically, goes to an owner or goes to the incident commander.
Key concepts
- Security data is sensitive in three ways
- It contains personal data, it describes weaknesses, and it describes the defender: suppression lists and alerts closed as benign tell an adversary what will not be seen. A leaked detection repository is worth more to an intruder than most production data.
- Six layers of a guardrail policy
- Input handling, retrieval scope, tools, output checks, budgets and evidence, each enforced at the policy point or by the platform and not by the model. Attacker-controlled fields are passed as delimited, labelled data, never concatenated into instructions.
- The ceiling
- Injection classifiers in the input layer reduce noise, but they are not the ceiling. The ceiling is set by the tools, the identity and the output checks, which hold whether or not a manipulation is recognised.
- Canary record
- A planted fictitious case whose identifiers exist nowhere else. Any appearance of those identifiers outside its boundary is a high-confidence signal of extraction through the AI service.
- Reversion per control class
- Drift in a reviewed class, such as a gateway bypass or egress beyond default-deny, is reverted automatically and reported; other drift goes to an owner with a deadline; drift during a declared incident goes to the incident commander.
Interactive exercises
Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.
Interactive exercise 1 · Match
Six layers of a guardrail policy
Match each layer of the guardrail policy to what it does for a security AI capability.
Interactive exercise 2 · Scenario
Drift during a declared incident
Apply the chapter's rule on what reverts automatically, as a platform engineer at Hansa Mutual.
Teaching plan
| 15 min | The teacher presents why security data is sensitive and the six data classes; participants identify where their own security AI has created new copies of such data. |
| 20 min | Walk through the six guardrail layers with the NR-SEC-001 policy: dropped fields, read-only tools, output schema and budgets. |
| 15 min | Review baselines and drift as policy as code, the three practices for soundness review, and provenance from commit trailer to verified signature. |
| 30 min | Exercise: groups apply a guardrail policy to one security AI capability using the nine steps of the practical guide. |
| 10 min | Groups name the worst outcome a manipulated input could still cause under their policy and which layer bounds it. |
Discussion
- The chapter says a guardrail switched off is a worse drift than most. Why, and how would your organisation notice within minutes?
- Which changes to security content in your team reduce what the defender sees, and who reviews them today?
- Reverting a rule an incident responder widened ten minutes ago can break containment. How should a declared incident change automatic reversion?
Written exercise
Exercise · 30 minutes
Apply a guardrail policy to a security AI capability
For a triage, enrichment or summarising capability in your organisation, or for Hansa Mutual's HM-SEC-001, write a guardrail policy. Record usage mode, autonomy level, identity and decision record. Mark every input field as trusted, organisation-written or attacker-controlled, treating doubtful fields as attacker-controlled, and state how such fields are handled. Agree a purpose allow-list and list fields to drop. Scope retrieval. Allow-list tools with read tools only at Level 0 and Level 1. Define the output schema and checks with the action on violation. Set budgets starting from the chapter's defaults. State what is logged, when the scenario library runs, and one chaos experiment with a marker phrase or canary record that would verify the policy.
Deliverable: A guardrail policy in the structure of NR-SEC-001-guardrails covering input, retrieval, tools, output, budgets, evidence and verification.
Self-test
Choose an answer to see whether it is right, or open the answer below each question.
What sets the ceiling on what a manipulated input can cause, according to the chapter?
Show the answer
A. The tools, the identity and the output checks
Classifiers reduce noise but are not the ceiling. Tools, identity and output checks hold whether or not a manipulation is recognised, because the controls that matter sit outside the model.Which security data does the chapter name as security-critical for backup under PR.DS-11, to be restorable to a known-good version?
Show the answer
C. Detection content, guardrail policies, model artefacts and evaluation sets
An adversary who alters detection content, guardrail policies, model artefacts or evaluation sets silently can blind the defender without touching production, so they must be restorable to a known-good version.Which changes require a second reviewer who did not prompt the assistant?
Show the answer
D. Changes that reduce what the defender sees, such as suppression lists and guardrail policies
The third practice for soundness review is that changes which reduce what the defender sees, such as suppression lists and guardrail policies, require a second reviewer who did not prompt the assistant.