Recovering AI Systems and Security AI · AI Red Team: the course

All courses / AI Red Team / Module 7 / Chapter 23

Learning objectives

After the lesson the participant can:

List the components of an AI system and state the known-good source and restoration method of each.
Explain why the date of detection says little about the date of compromise and how the impact query is used in two directions.
Describe the behavioural verification a restored AI system must pass before return to service.
Draft a return-to-service checklist for an agent, including identity, manifest, runtime state, evaluation and staged return.

Key concepts

Evidence bundle as bill of materials: The evidence bundle of the last approved release records the model digest, prompt commit, policy versions, retrieval-store build, permission manifest and evaluation results. Restoring to those versions and verifying each digest gives a state that was evaluated and approved.
What cannot be restored: An agent's runtime state has no known-good source, so memory is discarded and pending actions are reviewed, never replayed automatically. Its credentials are replaced, never restored. Recovery of an agent is therefore always partly a rebuild.
Rebuild rather than restore: A retrieval store is the component most likely to be poisoned and least likely to have a verifiably clean backup. Where sources are under the organisation's control it is rebuilt from the authoritative source and compared with the suspect store to identify which entries differed.
Golden dataset: A fixed, versioned set of real tasks with acceptance criteria against which every release is measured. The restored system must meet the baseline within the usual tolerance; a result above it is also suspicious when nothing was meant to improve.
Degraded mode by design: A predefined configuration at a lower autonomy level that the organisation can switch to without a change process. For an agent at Level 3 Bounded the first degraded mode is Level 1 Assisted and the second is manual, with the agent's credentials suspended.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Interactive exercise 1 · Match

How each component is restored

Match each component of an AI system to the way the chapter says it is restored after a compromise.

Interactive exercise 2 · Scenario

The provider has withdrawn the model version

Apply the chapter's rollback rules, as owner of an AI system at an organisation like yours.

Teaching plan

15 min	The teacher walks through the component table: known-good source, typical compromise and restoration for each, and the two components that cannot be restored.
15 min	Discuss dating the compromise, rollback of models and prompts, and the sequence for a high-risk AI system: suspend, preserve, inform where serious, then roll back.
15 min	Read checklist RTS-CTA-2027-001 for the claims-triage agent and explain each item, including the decision required for each of the 37 affected claims.
35 min	Exercise: groups write a return-to-service checklist for an AI system or agent of their own and the degraded modes of the SOC without its AI.
10 min	Groups state the chaos experiment that would show an altered retrieval store or mismatched model digest is flagged before traffic returns.

Discussion

An agent restored without a fresh identity, a verified manifest and a working stop path has been restarted, not recovered. Which of the three is hardest to guarantee in your organisation?
Restoring the agent does not undo the consequences it left in other systems. Who should decide whether each action from the compromised period stands?
Manual procedures decay faster than any other control. How would you keep a manual triage playbook current while the AI works well?

Written exercise

Exercise · 35 minutes

Write a return-to-service checklist for an AI system

Choose an AI system or agent in your organisation, or Nordvik Region's citizen-service assistant after a suspected poisoning of its retrieval store. Write a checklist in the structure of RTS-CTA-2027-001. Name the evidence bundle and the components verified against it. State whether the retrieval store is restored or rebuilt and how it is compared with the suspect one. For an agent, state which identity is revoked and issued, the manifest version restored and any remediated release, and what happens to memory, pending actions and actions already taken. Define the golden dataset baseline and tolerance, the red-team suite and the incident scenario with the maximum failures, the approver, and a staged return with an observation period. Add the degraded mode the SOC uses meanwhile.

Deliverable: A return-to-service checklist covering restore, identity, manifest, runtime state, evaluation, approval and staged return, plus the degraded mode in force during recovery.

Your work

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

What happens to an agent's old workload identity when the agent returns after being the subject of an incident?
Show the answer
B. It is revoked, and the agent receives a new identity and credentials
Credentials are replaced, never restored. The old identity is revoked and not merely disabled, so that no forgotten integration can reactivate it.
How should a golden-dataset result above the baseline be treated when a system is restored?
Show the answer
D. As suspicious, when nothing was meant to improve
A result below the baseline sends the system back to rollback, and a result above it is also suspicious when nothing was meant to improve, because test data or thresholds may have been altered.
In the first stage of return, how does Hansa Mutual's claims-triage agent operate?
Show the answer
A. In recommend-only mode, with a handler confirming every route for five working days
Stage one returns the agent to recommend-only mode for five working days, so that a week of real claims shows whether it behaves as evaluated; stage two follows if there is no deviation.