Incident Management and Analysis with AI

All courses / AI Red Team / Module 6 / Chapter 20

Learning objectives

After the lesson the participant can:

• Explain why declaration criteria are written, versioned and tested before any incident, and why early declaration is preferred.
• List the five records the chain of custody keeps for every AI contribution that informs a finding.
• Classify the role of an AI system in an incident as target, channel, subject or instrument of analysis.
• Design a tabletop in which declaration, the AI classification step and a seeded AI error are tested against agreed times.

Key concepts

Declaration criteria

Criteria written down in advance, such as any tool call by a production agent outside its permission manifest, against which AI tests the assembled case before paging the person who holds the decision with a proposed severity.

Impact query

A scoping question turned into a query that can be re-run and shown to someone else, versioned with the system. Its parameters are bound from the incident record, so the AI analysis service can propose a window but cannot change what the query does.

Chain of custody for AI analysis

For every AI contribution the record keeps the input data with its query and hash, the instruction, the model identifier and version, the full unedited output, and the human judgement. Original evidence is hashed on acquisition and never processed by a model directly.

AI classification step

A step placed at declaration that records which AI systems are involved and in which roles, their usage mode, whether the organisation is provider or deployer, whether any is high-risk and which regimes may apply. It does not decide whether to report; it brings the deciders in early.

The subject does not investigate itself

When an agent is the subject, it is stopped through the stop path and preserved, evidence comes from outside the agent, and if the subject is the organisation's own security AI the SOC runs in its degraded mode for the duration.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. Why does the chapter insist that AI must not become the record of decisions, even though it keeps the running log?
2. If the model's interpretation is lost, the finding still stands. What has to be true of your analysis practice for that sentence to hold?
3. How would a lightweight class of agent incident, handled in hours by one analyst and reviewed monthly, change how your organisation notices trends?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. When is an AI-assisted finding reproducible, according to the chapter?

   1. When the same model returns the same words for the same prompt
   2. When another analyst, given the recorded queries and data, reaches the same conclusion
   3. When two different models independently draft the same timeline
   4. When the vendor certifies the analysis service as deterministic and the model version is pinned in the record
   Show the answer

   B. When another analyst, given the recorded queries and data, reaches the same conclusion
   Models do not reliably produce the same output twice. The defensible structure is layered: deterministic queries produce data that can be re-run and hashed, the model proposes, and an analyst verifies and signs each finding.

2. What is the first rule when a Mode 5 agent is the subject of an incident?

   1. Ask the agent to summarise what it did before anything else
   2. Instruct the agent through its prompt to stop all pending actions and to report what it has queued
   3. Restart the agent from its known-good definition at once
   4. Stop it through the stop path by revoking credentials outside its runtime, then preserve
   Show the answer

   D. Stop it through the stop path by revoking credentials outside its runtime, then preserve
   Rule one is to revoke the agent's credentials outside its runtime and not to instruct it to stop, then preserve memory, queued actions and history before any restart. The agent's own account is at most a lead.

3. Which decisions does the chapter keep with people at every autonomy level?

   1. Opening the incident record and paging the on-call roles
   2. Starting a reversible log-retention hold on affected systems
   3. Declaration, severity, classification and findings
   4. Normalising log sources and correlating events across them
   Show the answer

   C. Declaration, severity, classification and findings
   The level table states that Level 4 Autonomous is not used: declaration, severity, classification and findings always rest with people. Opening the record, paging and reversible evidence preservation may run at Level 3 Bounded.