Introduction: Are the Dystopias Coming True?

All courses / AI Threat / Module 1 / Chapter 1

Learning objectives

After the lesson the participant can:

• Explain why the book treats the dystopian claim as a hypothesis to investigate and rejects both belief and dismissal.
• Classify an AI threat by scale: the human mind, society and truth, humans wielding machines, or machines beyond control.
• Distinguish capability, propensity and realised harm in a reported observation.
• Draft terms of reference for an AI threat assessment that state question, scope, method and deliverables.

Key concepts

The four scales of threat

The book orders the dystopias from the individual to civilisation: the human mind, society and truth, humans wielding machines, and machines beyond control. Evidence and scope for action differ at each scale, and the scales interact.

Evidence ladder

Five rungs that show what support a claim has: fiction, argument, laboratory demonstration, incident in the wild and pattern at scale. The rung does not make a claim true; it shows what would move it.

Capability, propensity, realised harm

A capability is what a system can do when fully elicited; a propensity is how likely it is to do it in the situations it meets; a realised harm is damage that has occurred.

Observable, expected, speculation

The series' outlook labels, carried over from AI Governance. A commitment can rest on what is observable, preparation on what is expected, and only reversible measures on what is speculative.

Constructive debate in four steps

Every debate is staged as the case for concern at its strongest, the case for calm with the same charity, where the positions meet, and what can be done. Only then comes the verdict.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. The chapter says belief leaves a board only one step, to stop using AI, which removes a small part of its exposure. Which parts of your organisation's exposure would a pause leave untouched, and why?
2. Why might a laboratory scenario designed to elicit blackmail be both important evidence and weak evidence, depending on what one takes it to show?
3. Nordvik risks answering too narrowly, as a cybersecurity question, and Hansa too technically. Which of the two errors is your own organisation more prone to, and what would correct it?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. According to the chapter, what does a laboratory demonstration of blackmail by a model in a simulated company establish?

   1. A propensity, because several developers' models showed it
   2. A capability, little about propensity and nothing about harm
   3. A realised harm, because an executive was threatened
   4. Nothing at all, because the scenario was designed
   Show the answer

   B. A capability, little about propensity and nothing about harm
   The chapter says such a demonstration establishes a capability, little about propensity and nothing about harm, whereas the July 2026 incident established all three in one unusual setting.

2. Which feature of Nordvik's terms of reference allows a pause to be weighed fairly against controls that cost money?

   1. The assessment is led by the security team alone
   2. National defence policy is placed inside the scope
   3. The report to the audit committee is kept classified
   4. Every recommended decision states its reversibility and cost
   Show the answer

   D. Every recommended decision states its reversibility and cost
   Because each decision states reversibility and cost, a pause, cheap to reverse but forgoing the benefits, can be compared with controls that cost money but keep them.

3. Why is Hansa Mutual described as exposed twice to AI threats?

   1. Through its own systems and through the cyber and liability policies it underwrites
   2. Through its staff assistant and through its citizen-service assistant
   3. Through European law and through the law of non-European states
   4. Through its red team and through its security operations centre
   Show the answer

   A. Through its own systems and through the cyber and liability policies it underwrites
   As an insurer, Hansa runs agents with write access to production and also underwrites cyber and liability policies, so AI-related losses can reach it from both sides.