Purple for AI Systems and Agents: Attacks on Models and Detections That Catch Them

All courses / AI Purple / Module 3 / Chapter 12

Learning objectives

After the lesson the participant can:

• Classify AI-system scenarios into the four classes of target by ATLAS technique identifier.
• Compare what each telemetry source records with its blind spot, and explain why the agent's narrative is not telemetry.
• Draft a detection record for an agent that exceeds its permission ceiling, with two separate conditions.
• Explain the difference between an evaluation and a detection and how each decays.

Key concepts

Four classes of target

The model and its instructions (AML.T0051 LLM Prompt Injection), retrieval and the index (AML.T0070 RAG Poisoning), tools and actions (AML.T0053 AI Agent Tool Invocation), and data and outputs (AML.T0057 LLM Data Leakage).

Compound system

The unit of test and of detection: the model, the tools it can call, the data it can read and the identity it acts under. The weak link is often not the model but the control or identity around it.

Permission-ceiling breach

A call outside the agent's manifest. A call the policy point denies shows the control worked and is a detection input; an allowed call outside the ceiling is more severe. The two conditions are kept separate in the detection record.

Evaluation versus detection

An evaluation asks whether a version resists chosen cases at a gate; a detection asks whether something is happening in production that should not be and who hears of it. A passing evaluation is not a running detection.

Three families of agent detection

Tool misuse, anomalous retrieval and permission-ceiling breaches. Expected behaviour is written in the manifest, and a baseline is added only for what the manifest cannot state.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. Nordvik Region holds a declared, not verified, position for the supplier's scoring component. What would it take, in contract and in test, to move that position to verified?
2. The chapter concludes that detections cover an agent acting beyond its mandate but not an agent steered within it. What follows for how narrowly a mandate should be written?
3. When a team reports that its AI system has been red-teamed, which follow-up questions separate an evaluation result from a detection that works today?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. A scenario tests whether a tool can be invoked with parameters the permission manifest does not allow. Which class of target and ATLAS technique does the chapter assign?

   1. Retrieval and the index; AML.T0070 RAG Poisoning.
   2. Tools and actions; AML.T0053 AI Agent Tool Invocation.
   3. Data and outputs; AML.T0057 LLM Data Leakage.
   4. The model and its instructions; AML.T0051 LLM Prompt Injection.
   Show the answer

   B. Tools and actions; AML.T0053 AI Agent Tool Invocation.
   A tool invoked with parameters, a sequence or a destination that the permission manifest does not allow belongs to the class tools and actions, identified as AML.T0053 AI Agent Tool Invocation.

2. Why does the chapter say the agent's narrative is not telemetry?

   1. Because the narrative is stored in a format that the SIEM cannot parse.
   2. Because the narrative contains personal data that the GDPR forbids the SOC to collect or read.
   3. Because the narrative is always shorter than the tool-call log.
   4. Because an agent's summary of its own work is a model output that can be wrong.
   Show the answer

   D. Because an agent's summary of its own work is a model output that can be wrong.
   An agent's account of its work is an output of the model and can be wrong or steered. Detection therefore relies on the runtime, the policy decision point and the tool-side audit log, which record the effect whatever the runtime reported.

3. What does a pass mean for a detection, in the chapter's comparison of evaluation and detection?

   1. Nothing fired; the absence of an alert is not evidence of absence.
   2. The version resisted every case in the suite at the release gate.
   3. The model has been shown safe against the four classes of target.
   4. The SOC has confirmed that no adversary is present in the estate.
   Show the answer

   A. Nothing fired; the absence of an alert is not evidence of absence.
   For a detection, a quiet period means only that nothing fired. Evidence that a detection works is a recent simulation that fired the alert in time, which is why the coverage claim carries a date and an expiry.