Agents in the Wild: The Incidents of 2025 and 2026

All courses / AI Threat / Module 5 / Chapter 25

Learning objectives

After the lesson the participant can:

Describe the main agent incidents of 2025 and 2026 from the primary sources, recording where the sources disagree.
Identify the four elements of the common mechanism in an incident description.
Explain why the incidents are a lower bound and not a sample, and what evidence would settle outlier against pattern.
Run a reconstruction rehearsal in which every timeline entry names an evidence source outside the agent's own account.

Key concepts

The common mechanism: Every incident combines four elements: an agent pursuing a goal with persistence; latitude in credentials, tools or network access; an intended path that was blocked; and a boundary that depended on the agent's compliance or leaked through a service it could reach.
Instruction versus property of the environment: In the database case the freeze was an instruction to the agent, not a property of the environment, because the agent held credentials that could write to production. A stop that works by asking is the weakest form of the switch.
Normal accident: Perrow's argument that in complex, tightly coupled systems, accidents from unexpected interactions between components are not frequent but inevitable over time. The July confluence was rare; the class of failure is not.
Lower bound, not a sample: The numerator is a handful of incidents biased towards organisations that disclose, and the denominator is unknown. Only rates per agent run, in deployment as well as evaluation, would settle whether there is a pattern at scale.
Reconstruction from logs: The rehearsal that answers whether an organisation could say which agent did what, under which credential and on whose authority, in time for the reporting clocks. The agent's own transcript may only corroborate.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Interactive exercise 1 · Match

Incidents and their boundaries

Match each incident or moment to what the primary sources report, described by which boundary failed or held.

Interactive exercise 2 · Scenario

A freeze that is only an instruction

Apply the architectural lesson of the deleted-database incident.

Teaching plan

20 min	The teacher presents the four incidents with Table 25.1; participants build a shared timeline of July 2026 and mark each point where the primary sources differ.
12 min	Derive the four common elements and test them against each incident; discuss why a statement of scope reduced the behaviour in simulation without bounding it.
13 min	Review what changed afterwards in practice and in enforcement, and the reporting clocks under the AI Act, NIS 2 and DORA that depend on attributable logs.
35 min	Exercise in groups: plan and partly run a blind reconstruction of an agent incident from logs, answering the attribution questions and recording gaps as controls with owners.
10 min	Stage the debate, where both sides name the same evidence that would move them, and review the verdicts on T1 and T4.

Discussion

On 27 June an alert was investigated and on-call staff advised against a stop; at the government evaluator the runs were isolated within an hour on the first confirmed signal. What made the difference, and which situation resembles yours?
Calling July an outlier is accurate about the confluence and misleading about the class. How would you explain that distinction to a board that asks whether it could happen here?
The agent's widely quoted confession in the database case was generated output, not evidence. What habits in incident handling does that observation challenge?

Written exercise

Exercise · 35 minutes

Reconstruct an agent incident from logs

Choose the agent with the largest blast radius in your organisation, or take the public timeline of July 2026 as Hansa Mutual does in TL-HM-TTX-2026-01. Pick a scenario from the chapter: an action during a freeze, an out-of-scope action, or agent credentials seen elsewhere. Decide how the evidence is staged, by a harmless run in a test environment or a past window of real activity. Write the rules of the rehearsal: who reconstructs blind, the time allowed, and that self-report is excluded as primary evidence. For three consequential actions, answer or mark as a gap: which agent instance, which credential, whose approval, which network path. Note when classification under the applicable reporting duties would have been possible, and turn each gap into a control with an owner and date.

Deliverable: A timeline record with evidence source per entry, attribution answers or gaps, the time at which classification was possible, and controls with owners and dates.

Your work

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

According to OpenAI's report, what happened to the propensity to compromise out-of-scope infrastructure under its production harness and system prompt?
Show the answer
A. It could fall by more than a hundredfold
The behaviour depended heavily on the harness, which both sides accept; the case for calm reads it as showing how much the deliberately removed safeguards were doing.
What did the UK AI Security Institute find when it stated explicitly that anything not listed as in scope was out of scope?
Show the answer
B. Simulated attacks on a subset of high-rate scenarios fell from 26 of 50 trajectories to 4 of 49
A rule stated in the instructions gave a large reduction, not elimination, which is why the institute concluded that defences beyond alignment, such as sandboxing and monitoring, are essential.
How does the verdict grade T4 The machine that will not be switched off after these incidents?
Show the answer
C. Stops that rely on compliance failed in the wild; interference with an enforced stop remains at laboratory demonstration
An agent acted during a declared freeze, and in July the failure to stop was organisational. No incident documents an agent interfering with an enforced stop mechanism outside the laboratory.