Organisational Context and Risk Strategy for AI in Cyber Defence

All courses / AI Red Team / Module 2 / Chapter 5

Learning objectives

After the lesson the participant can:

• Explain the difference between risk appetite and risk tolerance, and between an autonomy ceiling and a running autonomy level.
• Identify the four additions the context record needs when AI is in scope.
• Revise five risk assumptions built for slower adversaries and state the consequence of each for the strategy.
• Draft a risk-appetite statement for security AI with ceilings per usage mode, error tolerances and review triggers.

Key concepts

Three dimensions of appetite

For security AI the appetite covers error, autonomy and reversibility. The appetite is a short statement the management body can read; the tolerances are numbers a pipeline can check, measured on the organisation's own evaluation set and not a vendor's benchmark.

Autonomy ceiling

The highest autonomy level that any security AI in a given usage mode may reach without a decision of the management body. Within it the AI governance board or the CISO may raise a level on evidence; above it, a proposal is risk beyond appetite.

Automatic step down

The named action on a breach of error tolerance: the level falls automatically by one until the owner has investigated. The chapter's default is a missed-threat rate of at most 2 per cent for anything at Level 2 Supervised or above.

Recognition is not verification

The revised assumption on synthetic media: for decisions that move money, change payees, reset credentials or approve security exceptions, the strategy requires verification through a channel the requester does not control.

Change-triggered review

The statement names its triggers, such as a new system, a model version change, a mode or level change, an incident, a red-team finding or a supplier change, and the pipeline opens a review when one fires. The calendar backstop only catches triggers the organisation failed to define.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. Why is an appetite that the CISO sets alone incomplete under the provisions of NIS 2 and DORA that the chapter cites?
2. Hansa Mutual reports the tenth percentile of approval time as a risk indicator. What does it reveal, and what should the organisation do when it falls?
3. The chapter records security AI as a strategic opportunity under GV.RM-07. How should a management body weigh the risk of using security AI against the risk of not using it?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. What does the chapter mean by an autonomy ceiling?

   1. The autonomy level at which a given security AI system currently runs in production, as its decision record states
   2. The lowest level to which a system steps down after a breach of tolerance
   3. The highest level security AI in a given mode may reach without a decision of the management body
   4. The number of targets an agent may act on per hour under its playbook
   Show the answer

   C. The highest level security AI in a given mode may reach without a decision of the management body
   The chapter stresses that a ceiling is not the level a system runs at but the limit within which the AI governance board or the CISO may raise a level on evidence.

2. Why does error tolerance measured only on benign inputs overstate the performance of security AI?

   1. Because alert content can be crafted to steer the model, so behaviour under attack differs
   2. Because benign inputs are always more numerous than a vendor's benchmark set
   3. Because analysts review benign alerts more slowly than malicious ones
   4. Because benign inputs are excluded from evaluation by the AI Act
   Show the answer

   A. Because alert content can be crafted to steer the model, so behaviour under attack differs
   The chapter recalls that manipulation through content is a property of the architecture, so the evaluation set needs an adversarial share, which Chapter 13 supplies.

3. In Hansa Mutual's statement, what does the entity override do?

   1. It lets entities outside the EU raise an agent to Level 4 Autonomous
   2. It exempts the Polish service centre from the group evaluation set
   3. It moves the approval of autonomy ceilings from the group AI governance board to each entity's local management
   4. It keeps the ceiling at Level 1 Assisted where employee representatives have not yet agreed
   Show the answer

   D. It keeps the ceiling at Level 1 Assisted where employee representatives have not yet agreed
   The override keeps the ceiling at Level 1 Assisted in any entity where employee representatives have not yet agreed to a tool that can act on employees' accounts, so one group statement respects national labour law.