Oversight and AI Supply Chain Risk

All courses / AI Red Team / Module 2 / Chapter 8

Learning objectives

After the lesson the participant can:

• Describe the three kinds of content an oversight report on security AI gives the management body.
• Determine the effective autonomy of a vendor AI feature and compare it with the ceiling for its action class.
• Distinguish what the provider, the vendor, the MSSP and the deployer each test, and what the deployer must require by contract.
• Draft supplier assurance requirements for security AI covering declaration, control, change notification, evidence, incident and exit.

Key concepts

Effective autonomy

The level at which a supplier feature actually acts, whatever its marketing calls it. Establishing it needs three facts: where AI is used, which actions it takes without per-action approval, and whether they can be reversed. Undeclared, it is a supply-chain finding in its own right.

Testing the compound system

The deployer evaluates each feature on its own evaluation set built from historical alerts with known outcomes, red teams features that read attacker-controlled content and verifies behaviour with chaos experiments. Provider evidence is input to model selection, never a substitute for own testing.

Canary set

A fixed set of benign, labelled samples, 20 to 50 in the practical guide, submitted on a schedule and compared with the baseline. It detects a change in behaviour even when the version string stays the same.

Exit plan for security AI

The realistic exit is often a fall-back to the process without AI and not a move to another vendor. The plan says how the SOC works without the feature tomorrow and what moves with the organisation; switching it off for a day in a controlled test turns assertion into evidence.

Change-triggered reassessment

Reassessment is due for a new model version, a new AI feature, a change in the supplier's set-up or a change in the threat. Until it is complete, the safe default for a feature that acts automatically is to fall back to proposals at Level 1 Assisted.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. Why does an adversarially evaluated general-purpose AI model not answer the deployer's question about a triage feature that reads alert fields?
2. A report that asks for nothing invites noting and not oversight. Which decisions on security AI would you put to your management body in the next quarter?
3. For a public authority the tender is the moment of greatest leverage. What would you insist on in a tender that is hard to add to a framework agreement afterwards?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. An email gateway removes a reported message from every mailbox on a classifier verdict, without an analyst approving. How does the chapter classify it?

   1. Level 1 Assisted, because the user reported the message first
   2. Level 3 Bounded or above, whatever its marketing calls the feature
   3. Level 2 Supervised, because the vendor supervises the model
   4. Level 0 Advisory, because the verdict is only a recommendation
   Show the answer

   B. Level 3 Bounded or above, whatever its marketing calls the feature
   The feature acts without per-action approval, so it acts at Level 3 Bounded or above; the vendor has chosen the level on the deployer's behalf, and the chapter calls this its effective autonomy.

2. Which of the following can the deployer not test itself and must therefore require by contract and accept as attestation?

   1. Whether verdicts on a fixed set of benign samples drift over time
   2. Whether the feature's actions are logged to the deployer's SIEM
   3. The feature's results on labelled historical alerts with known outcomes
   4. The vendor's secure development process
   Show the answer

   D. The vendor's secure development process
   The chapter lists the provider's training data, the vendor's secure development process and the MSSP's internal access controls as things the deployer cannot test and must require by contract.

3. What is the cheapest technical signal for supplier model changes, which the chapter says should be required by contract?

   1. Version telemetry, where the product reports the model version behind each verdict
   2. Verdict-distribution monitoring on all live traffic, which catches every change with the least noise
   3. A quarterly attestation from the model provider that no model version has changed
   4. Manual comparison of the vendor's release notes with the features enabled in the console
   Show the answer

   A. Version telemetry, where the product reports the model version behind each verdict
   Version telemetry is named as the cheapest signal. The canary set catches behaviour changes behind an unchanged version string, and verdict-distribution monitoring catches what the canary set misses at the cost of more noise.