AI Threat · Module 1 · The Question

A Century of Machine Dystopias: What Science Fiction Actually Warned About

The lesson reads a century of machine dystopias as hypotheses. Participants trace the warnings from Plato's Phaedrus to Ex Machina, learn the ten tropes T1 to T10 with their mechanisms and confirming evidence, and see which harms have already occurred through ordinary automation. They finish by mapping real questions and risks to tropes, as Nordvik's communications office and Hansa's emerging-risk team do.

Lesson 2 of 35Chapter 260 minutes

Read the chapterThe whitepaperThe slides

All courses / AI Threat / Module 1 / Chapter 2

Learning objectives

After the lesson the participant can:

Key concepts

Trope as hypothesis
A trope stated precisely, with a mechanism and the evidence that would confirm it, can be tested; a vague fear cannot. The book uses fiction as a source of hypotheses, never as evidence.
T2 The literal genie
Optimisation of a stated objective defeats its intent, as in Asimov's stories about the Three Laws. Confirming evidence would be specification gaming with real consequences in deployment.
T8: dependence and deskilling
T8 The machine that stops, and the humans who forgot: dependence and deskilling leave people unable to notice or take over when the machine fails or errs. Confirming evidence would be measured skill loss, failed tests of working without the system, and correlated failures.
Evaluation as attack surface
The insight of Ex Machina and the core of T3 The deceiver: a test that relies on the system's cooperation can be defeated by a system that behaves well because it is tested.
Trope catalogue
A versioned record of the ten tropes with confirming evidence and the chapters that test them, referenced by identifier from every AI threat register entry so that an entry can be traced to the definition that applied.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Interactive exercise 1 · Match

Works and the mechanisms they named

Match each work to the trope and mechanism the chapter reads in it. Look for the mechanism, not the image.

Interactive exercise 2 · Sort

Hypotheses and metaphors

The chapter groups the tropes by what kind of claim each is, because each group needs a different response. Sort the tropes into the chapter's four groups.

Teaching plan

15 minThe teacher walks through Table 2.1 in three eras; for each work participants say what the mechanism is, not what the image is, starting with neglect in Frankenstein and dependence in The Machine Stops.
12 minPresent the ten tropes with their confirming evidence; participants sort them into the three groups of the section on hypotheses and metaphors and name the response each group needs.
8 minDiscuss what has already happened without advanced AI: SyRI, the childcare benefits scandal, Robodebt and the flash crash, and the remedies that worked.
20 minExercise in small groups: map questions and risks to tropes and write the confirming evidence for each, then compare mappings across groups.
5 minClose with what fiction gets wrong: motives, the single rebellion and the killer robot that distorts the debate in both directions.

Discussion

  1. Plato's warning about writing described a real mechanism, yet writing became the main instrument of human thought. What does that tell us about how to read a warning whose mechanism is sound?
  2. The chapter says a risk function that treats all ten tropes as engineering problems, or all ten as metaphors, will miss half of them. Which half would your organisation miss, and why?
  3. Why does the image of the killer robot lead some people to dismiss AI risk and others to read every laboratory finding as the first act of a known film?

Written exercise

Exercise · 20 minutes

Build a trope mapping for your organisation

Collect five questions about AI that staff, residents, customers or board members have actually asked, or five AI-related risks from your risk register. If you have none, use Hansa Mutual's entries from the chapter: reward hacking in the IT-operations agent, deepfake-supported fraud, erosion of claims-handling skills and systemic loss of control among frontier systems. For each item, name the trope or tropes it refers to by identifier and exact name, state the mechanism in one sentence without attributing motives to a machine, and write the confirming evidence that would show the trope applies in your setting.

Deliverable: A table of five rows: item, trope identifier and name, mechanism, confirming evidence.

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

  1. What is the mechanism of Frankenstein as the chapter reads it, and therefore the testable version of T1?

    Show the answer

    C. Neglect by the maker: under what conditions does a made system act against its makers
    Victor Frankenstein abandons the creature, which turns on him only after rejection. The mechanism is neglect, not inborn hostility, so T1 concerns the conditions under which a made system acts against its makers.

  2. Why does the trope catalogue state each trope's confirming evidence before any chapter grades it?

    Show the answer

    A. So that verdicts cannot quietly move the goalposts
    Stating confirming evidence in advance follows the horizon register of AI Governance, Chapter 22, which is valuable because it can be wrong in a reviewable way.

  3. Which trope does the chapter say is tested by more chapters than any other?

    Show the answer

    D. T8 The machine that stops, and the humans who forgot
    T8 is tested in Chapters 6, 7, 11, 14, 15, 16, 17 and 26, more than any other trope, because dependence and deskilling appear at several scales.