When It Breaks: Incidents, Outages and AI-Generated Defects

All courses / AI Developer / Module 4 / Chapter 23

Learning objectives

After the lesson the participant can:

Classify an event as a defect in generated code, an agent action beyond intent, a toolchain incident or a provider-side change
Explain how blameless review and the Name test answer different questions and work together
Classify the mechanism of a failure using the five mechanism classes and name the control that belongs in the process
Draft the AI-involvement fields of a post-incident review, including unknown as a value

Key concepts

The additional questions: Who owned the change? What did the tool do, and can anyone say? Was the oracle independent of the generator? Who could stop the change, and how long did it take? They are added to the usual incident method, not substituted for it.
Blameless and named: Blame asks who deserves a sanction; naming asks who owns the change. The record names roles and conditions, not culprits, the three tests become questions about the system, and the review protects the person nearest the system from becoming a moral crumple zone.
The model did it: Not a contributing condition, because a model holds no role in the change process. The review looks for conditions in the path: the oracle, the specification, the release path, the mandate and the toolchain.
Five mechanism classes: Shared blind spot, specification gap, unsafe release path, excess of agency, and toolchain or provider change. The first two concern the organisation's own proof and are the most common in a development organisation.
Unknown as a value: AI involvement is often unknowable, especially for ambient completion. The review field records unknown or unknown_partly with a consequence field, and the register reports the share of reviews with unknown involvement each quarter.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Interactive exercise 1 · Match

Mechanism classes and what the review looks for

Match each mechanism class of AI-involved failure to what the post-incident review looks for.

Interactive exercise 2 · Scenario

The draft review says the model did it

Apply the chapter's reasoning on blameless and named review.

Teaching plan

15 min	Lecture on the four kinds of event and the four stages of practice: stop first and name early, triggers, accounts before the timeline, and actions that close by evidence.
20 min	Case reading in groups: the form change Nordvik Region rolled back and the item Hansa Mutual had listed as not covered. Each group names the mechanism class and the result of the three tests.
15 min	The teacher presents Table 23.2 of verified public incidents, stressing what each primary source supports and that the agent case is cited only as reported.
30 min	Exercise: groups add AI-involvement fields to a post-incident review template and fill them for a past incident.
10 min	Short input on the reporting clocks and who classifies; the developer supplies facts and does not own the classification.

Discussion

In both illustrative incidents a gap was known on paper and nothing connected it to an owner and a date. Where do such gaps sit in your own records right now?
CrowdStrike and Knight Capital involved no AI. Why does the chapter give them so much space in a book about developers and AI?
The review record contains names and sometimes traces of individual work. What must be agreed with representatives before the template is adopted?

Written exercise

Exercise · 30 minutes

AI-involvement fields for a post-incident review

Take a past incident from your organisation, or the home-care form change at Nordvik Region. Extend the review template with a block for AI involvement with the values none, suggested, drafted, executed, unknown and unknown_partly, and a consequence for not knowing. List the evidence that exists for each tool and what is not kept. Add the oracle field: who wrote it, from what source, and whether it could see the failing case. Add the three stewardship tests with a result and a time each, and the measured time from decision to revert. Add the mechanism class, a statement that no individual is named as a cause, and the reporting block. Then fill the fields and count the unknowns.

Deliverable: An extended review template filled in for one incident, with the number of unknown fields counted and dated.

Your work

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

What did the review PIR-NR-2030-0006 of the Nordvik Region form change find?
Show the answer
C. A specification gap, not a model fault; Understand failed for the mapping
The interface ignored a new flag that the specification had recorded as an assumption with no acceptance criterion. Nobody could say which lines were generated and it did not matter; the owner could explain the form change but not the mapping.
How long did the revert take from the decision in the Hansa Mutual incident, and what did the steward's result show?
Show the answer
B. 38 minutes, and the steward passed Name, Understand and Revert
Stefan Brückner was named in the verification plan, restated the rule from the wording, and the rule was reverted in 38 minutes with authority held outside the pipeline. The 22 minutes belong to the Nordvik incident.
Under NIS 2 Article 23(4), as the chapter's table gives it, what are the first two clocks for a significant incident?
Show the answer
D. Early warning within 24 hours of becoming aware and incident notification within 72 hours
The table gives an early warning within 24 hours of becoming aware and an incident notification within 72 hours for essential and important entities. The clocks run from awareness, not from the end of the analysis.