Threat Modelling Transformed: Continuous, Generated and AI-Assisted

All courses / The Transformed Security Architect / Module 2 / Chapter 6

Learning objectives

After the lesson the participant can:

• Explain why a regenerated threat model can show what an annual workshop cannot, and what the workshop is still for.
• Identify the six recurring trust boundaries of an AI system and the legs of the lethal trifecta an agent holds.
• Decide candidate threats as accepted, reclassified, rejected or deferred against the soundness criteria, recording a reason.
• List the conditions under which the threat-model gate fails a merge.

Key concepts

Two layers of generation

A deterministic layer emits a boundary record and STRIDE threat stubs for every relationship crossing a trust zone and compares them with the last accepted model; an AI layer adds candidates the rules cannot derive. Each threat keeps an identifier that survives regeneration.

Lethal trifecta

Simon Willison's test for agentic systems: an agent with access to private data, exposure to untrusted content and the ability to communicate externally can be steered into disclosing the data. The model records which legs are present and which control removes the others.

Vector as a class of behaviour

A threat entry records the vector as a class of behaviour and never as an operational recipe. That is enough to design a control, a detection and a test, and keeps the model shareable with a supplier, an auditor and a works council.

Reviewer decisions

A named person accepts each candidate, reclassifies it, rejects it with a reason and a revisitWhen trigger, or defers it to an owner and a date after which the pipeline blocks. Accepting residual risk is a separate decision for the risk owner.

verifiedBy

The field that points an accepted threat to an attack case described by its set-up and expected outcome. A threat without one is declared, not verified, because an accepted threat that nothing tests is a belief.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. The gate rule that fails when a boundary changed without any threat considered is called the most valuable. Why can a reviewer who reads only candidates not see that failure?
2. A long list of fluent candidates invites complacency and automation bias. What in the design of the review loop would you change to counter that in your own team?
3. Nordvik models a few systems in depth because its reviewers are few; Hansa needs generation because no person can read every pull request. Which constraint governs your choice of method?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. At which autonomy level does the chapter place the AI generator of candidate threats by default?

   1. Level 0 Advisory, with no access to the repository
   2. Level 1 Assisted, writing only to a pull request comment
   3. Level 2 Supervised, writing accepted items directly
   4. Level 3 Bounded, accepting low-risk threats within a playbook
   Show the answer

   B. Level 1 Assisted, writing only to a pull request comment
   The generator works at Level 1 Assisted, writes only to a pull request comment and never to the accepted model, and no AI accepts a threat, a risk or an exception.

2. In Hansa's threat model, why is T-103, claim data leaving through an outbound channel, rejected?

   1. Because the threat-modeller agent classified it under the wrong STRIDE category
   2. Because the risk owner accepted the residual risk with an expiry date
   3. Because the attack case in HM-SEC-003 passed in the last campaign
   4. Because the manifest holds no outbound tool, and revisitWhen brings it back if one is added
   Show the answer

   D. Because the manifest holds no outbound tool, and revisitWhen brings it back if one is added
   The architecture cuts the third leg of the trifecta by denying egress and the manifest has no outbound tool, so T-103 is rejected with the trigger revisitWhen: tools, which raises it again the day an outbound tool is added.

3. Which mitigation does the soundness criterion on the right family explicitly rule out as a replacement for a control?

   1. Improving the system prompt in place of a control outside the model
   2. Entitlement filtering on the retrieval index
   3. Task-scoped credentials for tool calls
   4. Schema validation of output before it is written into a system of record
   Show the answer

   A. Improving the system prompt in place of a control outside the model
   A candidate must be mitigated by the right family, never 'improve the system prompt' in place of a control outside the model, because prompt injection is a property of the architecture and the controls sit outside the model.