AI Security Engineering in Practice

All courses / AI Platform and Security / Module 4 / Chapter 17

Learning objectives

After the lesson the participant can:

• Apply the threat-modelling method: six asset classes, trust boundaries and STRIDE extended with ATLAS and OWASP.
• Describe the five controls that secure retrieval and explain why entitlements are filtered before similarity search.
• Design a security test stage for the pipeline with explicit pass thresholds per suite.
• Draft an incident playbook that runs containment in parallel with legal classification.

Key concepts

Defence in depth for AI

Layers from the data and training path through retrieval and the gateway to agents and tools, each assuming the one before it has failed. The threat model, the security tests in CI and the tamper-evident audit log cut across all of them.

Six AI asset classes

Training data; model weights and adapters; prompts, which encode business rules; the retrieval corpus and its index; tools and their credentials; and outputs, which become actions and records. Each has a classification and an owner.

Review trigger

The list in the threat model of configuration paths, such as retrieval sources, tools, model version and prompt template, whose change forces a review. A code-ownership rule routes such changes to the security architect.

Entitlement-aware retrieval

Each document's access attributes are carried onto every chunk, and the filter on the user's delegated identity is applied before the similarity search, never after, because a post-filter leaks through ranking.

Control catalogue entry

An entry that carries framework mappings, the configuration key that implements the control and an evidence query with a fail condition. The pipeline resolves the implementation path, so a renamed configuration fails the build.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. The security test job tolerates a small success rate for injection attempts but fails on any leakage of another customer's data. How would you justify different thresholds for different suites in your own systems?
2. The record-keeping duty of AI Act Article 12 and the GDPR's data minimisation principle pull in opposite directions. How does logging identifiers and hashes resolve that, and what does it require of your stores?
3. Which of your controls is falsifiable today, in the sense that a query would show it failing?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. Why must the entitlement filter be applied before the similarity search and never after?

   1. Because filtering afterwards doubles the cost of the embedding model
   2. Because the gateway cannot read entitlements once chunks are ranked
   3. Because a post-filter leaks through ranking
   4. Because the AI Act prescribes the order of retrieval steps
   Show the answer

   C. Because a post-filter leaks through ranking
   The chapter requires each chunk to carry its document's access attributes and the filter on the delegated identity to run before search, since filtering afterwards leaks through ranking.

2. In the agent authorisation policy, which two properties does the chapter say matter more than the details?

   1. An approval is bound to the hash of the specific action, and the token's subject must equal the user
   2. The policy is written in Rego, and it is evaluated by Open Policy Agent
   3. Refunds are capped at a fixed amount, and approvals never expire
   4. The agent holds a long-lived key, and the operator can edit the log
   Show the answer

   A. An approval is bound to the hash of the specific action, and the token's subject must equal the user
   Approving one refund cannot authorise the next, and a hijacked agent cannot act for a customer who never opened the session.

3. How does the chapter position red teaming in the delivery of an AI system?

   1. As a single exercise before launch that need not be repeated
   2. As external discovery whose every finding becomes an automated test that also runs on a schedule
   3. As a replacement for threat-led penetration testing under DORA
   4. As a task for the model provider alone under Article 55, with the documentation requested in procurement
   Show the answer

   B. As external discovery whose every finding becomes an automated test that also runs on a schedule
   Behaviour changes with every prompt edit and model update, so findings become version-controlled suites in CI, run weekly as well to catch drift in a third-party model.