Security for AI Systems

All courses / AI Platform and Security / Module 4 / Chapter 16

Learning objectives

After the lesson the participant can:

• Explain the three properties of AI systems that break long-standing security assumptions.
• Map the points of the threat surface of an AI system to their main threats and primary architectural controls.
• Describe how an ML-BOM, signing and provenance bring a model into an existing promotion gate.
• Draft a threat model entry whose controls resolve to a guardrail configuration.

Key concepts

Single channel for instructions and data

A large language model receives its system prompt, the user's request and any retrieved document as one sequence of tokens, and nothing in the model enforces a boundary between them. The boundary is therefore enforced outside the model.

Indirect prompt injection

Instructions planted in content such as a web page, an email or a shared document that are followed when the content enters the context window. The attacker never touches the application, so every system that reads external content is exposed by construction.

ML-BOM

The CycloneDX Machine Learning Bill of Materials, which records the provenance, licences and training parameters of models and datasets. When a base model is withdrawn, a query over the registry's ML-BOMs lists every model that depends on it.

Threat model as code

A threat model kept in the repository with trust boundaries, threats tagged with OWASP and ATLAS identifiers, and named controls. It is reviewed on every change, because the pull request that adds a source or tool shows the entries affected.

Guardrail policy

The configuration the AI gateway enforces at runtime: allowed retrieval collections, output validation, tool allow-list, actions to confirm and limits. The threat model's control identifiers point to it, so a named control is demonstrably live.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. The checklist puts foundations first because a reviewer who starts with the AI-specific items tends to miss a shared API key. Which foundation is weakest around your AI systems?
2. None of the controls makes injection impossible; together they make it unprofitable. What is the worst case of a fully hijacked model in one of your systems today?
3. Could your organisation establish within 24 hours whether retrieved content caused a wrongful action? What would have to be logged for that?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. Why is indirect prompt injection described as more dangerous for enterprises than direct injection?

   1. Because it requires the attacker to hold valid credentials and a delegated token for the application's gateway route
   2. Because it only affects models that were fine-tuned on raw personal data
   3. Because it can be carried out only against self-hosted models
   4. Because the attacker never touches the application: planted content acts when it enters the context
   Show the answer

   D. Because the attacker never touches the application: planted content acts when it enters the context
   Instructions planted in a page, an email or a shared document act when retrieved, so every retrieval system and agent that reads external content is exposed by construction.

2. What does the chapter say is the property that matters in a codified threat model and guardrail policy?

   1. Completeness: every OWASP entry and every conceivable attack wording is enumerated in the model before the first release
   2. Linkage: the threat names the control, the control names the configuration, the pipeline proves deployment
   3. Confidentiality: the threat model is kept outside the repository
   4. Brevity: no more than ten threats are recorded for each system
   Show the answer

   B. Linkage: the threat names the control, the control names the configuration, the pipeline proves deployment
   The chapter states that the property that matters is linkage, not completeness, so that a named control is demonstrably live.

3. In the casework assistant's prompt section on untrusted content, what does the rule that the prompt contains no credentials record?

   1. That no control depends on the prompt's confidentiality; the case management API decides authorisation
   2. That the prompt may be published unredacted in the public algorithm register alongside the threat model
   3. That the model is not permitted to call any tool at all
   4. That the gateway does not need to log the prompt version
   Show the answer

   A. That no control depends on the prompt's confidentiality; the case management API decides authorisation
   The rule addresses system prompt leakage: the prompt enforces no boundary and holds no secrets or authorisation logic.