All courses / The Transformed Security Architect / Module 3 / Chapter 12
Learning objectives
After the lesson the participant can:
- Match each of the four kinds of assisted task to its characteristic failure and the verification that suits it.
- Summarise what the cited studies found on productivity and on quality, and what the chapter labels observable and expected.
- Design a routing rule for architecture information by class that is enforced outside the client.
- Apply the three tests that decide when AI-assisted work needs a decision record.
Key concepts
- Four kinds of task
- Drafting fails as fluent text stating an unreached position, research as a fabricated or misattributed source, review as an invented or missed finding, and analysis as a pattern from an unrepresentative sample. The assistant is used where verification costs less than doing the work unaided.
- Jagged frontier
- The finding that consultants did better on tasks inside what the model could do and were 19 percentage points less likely to reach a correct answer on a task outside it. A plausible answer looks the same on both sides.
- Protected oracles
- Tests, policy-as-code rules and schemas kept in a path the tool cannot write to, because an agentic tool asked to make a failing test pass may change the test. A change that touches an oracle is flagged for review.
- Routing by class
- Public and internal work may use the approved enterprise tier of a cloud assistant; confidential and protected work reaches only self-hosted or sovereign-hosted models through the AI gateway from a workspace with default-deny egress. Unlabelled work defaults to confidential.
- Work log, session record, decision record
- Three records with distinct purposes: the work log says which drafts came from the tool and how they were verified; the session record keeps full prompts and outputs for a short fixed period; the decision record says why the architect decided as they did.
Interactive exercises
Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.
Interactive exercise 1 · Match
Four tasks, four verifications
Match each kind of task an assistant can take part in to the verification the chapter says suits its characteristic failure.
Interactive exercise 2 · Sort
Routing architecture work by class
Sort each piece of work under the route the chapter's routing rule allows for it.
Teaching plan
| 15 min | Lecture on the four kinds of task and Modes 1 to 3 for architecture work; participants list their own current uses and the mode of each. |
| 15 min | The teacher presents the studies on productivity and on quality; participants sort each finding as a gain, a loss or a warning about the reviewer's confidence. |
| 10 min | Read MND-NR-01 together: find the autonomy per action, the denied actions, the protected oracle paths and the stop path at the gateway. |
| 25 min | Exercise: participants write a routing rule and a mandate manifest for their own Mode 3 use, and start a work log. |
| 10 min | Debrief on the five habits and on the three tests for a decision record, applied to one recent decision from each participant. |
Discussion
- Developers in the randomised trial predicted and afterwards still believed in a speed-up they did not have. How would you measure the effect of assistants on your own architecture work?
- Self-hosting buys confidentiality at the price of a verification burden. Where would you draw the line between the cloud tier and the self-hosted route for your architecture information, and why?
- The chapter says an architect who describes a design with evident confidence receives an answer shaped by that confidence. What changes in how you ask would counter that?
Written exercise
Exercise · 25 minutes
A personal AI workflow for architecture work
Following the chapter's practical guide, set up your own workflow on paper. Inventory the AI you already use in Modes 0 to 3, with the identity, reachable data and provider of each. Classify the architecture information you handle as public, internal, confidential or protected, and write the approved route for each class. Then write a mandate manifest for one Mode 3 tool: holder and approver, allowed models and route, allowed data classes and the three things that never leave the controlled route, autonomy per action with no more than Level 2 Supervised for edits, allowed tools, protected oracle paths, records and a stop path outside the tool. Add the first five lines of a work log.
Deliverable: A routing table by class, a mandate manifest in the form of MND-NR-01 and a five-line work log stating what was drafted with the tool and how it was verified.
Self-test
Choose an answer to see whether it is right, or open the answer below each question.
What did the randomised trial with sixteen experienced open-source developers on 246 real issues find?
Show the answer
C. Tasks took 19 per cent longer with AI, though the developers believed they had been sped up
The tasks took 19 per cent longer when AI was allowed, although the developers had predicted a 24 per cent speed-up and afterwards still believed in about 20 per cent. Feeling helped is not a measurement.In the mandate manifest MND-NR-01, version 1.0.0, what may the Mode 3 tool do with commits?
Show the answer
A. Nothing: commit and push are denied, while edits and commands run at Level 2 Supervised
The manifest sets edit_files and run_commands at Level 2 Supervised, where the architect approves each, and denies commit_and_push. The tool also cannot write to the protected oracle paths.According to the chapter, when does a decision that an assistant shaped materially need a decision record?
Show the answer
D. When options, framing or evidence came from the assistant and the decision is hard to reverse
The second test requires a record when the options, framing or evidence came from the assistant and the decision is hard to reverse, such as a boundary placed, an exception granted or a tender requirement written.