AI-Assisted Threat Modelling and Risk Assessment

All courses / AI Red Team / Module 3 / Chapter 10

Learning objectives

After the lesson the participant can:

• Explain why a threat model drawn once in a workshop fails against the obligations and threats the chapter describes.
• Classify a threat at a trust boundary by STRIDE category, ATT&CK or ATLAS technique and OWASP identifier.
• Apply the seven soundness checks to an AI-proposed threat and its score.
• Review a threat-model change on a pull request, recording an accept or reject decision with a reason for each candidate.

Key concepts

Candidate threats

AI proposes threats for every pull request that touches a trigger path and shows only what differs from the accepted model. A candidate joins the model only when a named person accepts it, and a rejected candidate stays rejected, with a reason, until the architecture changes.

Mitigation family

Because prompt injection is a property of the architecture, a candidate that proposes improving the system prompt as mitigation is unsound. Least privilege for tools, allow-listed retrieval, output validation or human approval are in the right family, outside the model.

Versioned assumptions

Adversary capability enters the risk model as a small set of named, versioned assumptions, such as time to exploit, the credibility of voice and video impersonation and the volume of targeted phishing, so that changing one rescores every dependent risk in a visible pull request.

AI edges in the attack graph

An agent holding a tool credential is an edge to everything the tool can change; a retrieval index ingesting external content is an edge from whoever can write the source to every prompt it feeds; a coding agent is an edge from anything it reads to everything its engineer can do.

Soundness checks

Grounding, classification, mitigation family, likelihood basis, impact basis, completeness and stability. The mechanical part is automated in the pipeline, a named reviewer judges against provenance, and the quality of that judgement is itself measured through sampled second reviews.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. A missed threat is invisible in the diff. Which of the seven checks addresses that, and how far can it be trusted?
2. Why should path-finding remain a deterministic computation while ranking may use AI, and what makes the ranking the most dangerous step?
3. How would treating time to exploit as a versioned assumption change your organisation's current risk register?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. In the worked example, why does the rejected candidate T-033 stay in the threat-model file?

   1. Because the generator cannot delete entries from an accepted model
   2. Because public-access rules require every candidate threat to be published together with the accepted model
   3. So the next regeneration does not raise it again and an auditor can see it was considered
   4. So that the risk owner can later accept it as residual risk
   Show the answer

   C. So the next regeneration does not raise it again and an auditor can see it was considered
   The chapter states that the rejected candidate stays with its reason, so the next regeneration does not raise it again for the same architecture state and an auditor can see that it was considered.

2. What may the threat-model generator write to, according to the chapter?

   1. Only a pull request comment, never the accepted model
   2. The accepted model, once the automated checks have passed
   3. The risk register, for threats below the appetite threshold
   4. The architecture definitions, to add missing trust boundaries
   Show the answer

   A. Only a pull request comment, never the accepted model
   The generator reads architecture definitions, the accepted model and pinned catalogues, runs on trusted infrastructure and writes only to a pull request comment; a named reviewer decides each candidate.

3. Which use does the chapter rule out at every autonomy level for AI in risk assessment?

   1. Drafting candidate threats and scores for a named reviewer
   2. Rescoring risks after an approved change to an assumption
   3. Ranking attack paths with explanations that can be checked
   4. Accepting a threat, a risk or an exception
   Show the answer

   D. Accepting a threat, a risk or an exception
   The level table states that Level 4 Autonomous is not used: no AI accepts a threat, a risk or an exception. Acceptance of residual risk is reserved to the risk owner.