Conclusions: Is There Anything in It? · AI Threat: the course

All courses / AI Threat / Module 7 / Chapter 35

Learning objectives

After the lesson the participant can:

State the verdict for each of the ten tropes: highest rung reached, outlook and degree of confirmation.
Explain the shape of the answer by scale: where evidence is strongest and where ordinary influence is greatest.
Distinguish what science fiction got right from the three things it got wrong.
Name the signals that would change the verdict, in both directions, and separate the author's judgement from the findings.

Key concepts

Not what the stories said: Most mechanisms science fiction named have been observed, several as incidents in the wild and two as patterns at scale, largely through ordinary automation. What has not been observed is the monster: hostility, self-improvement beyond control or the end of the human story.
Narrow form and full form: Each verdict separates the rung of the narrow, tested form from the full fictional form, so that incident in the wild is not mistaken for the film having started. No trope is refuted and none is confirmed in its fictional form.
The shape of the answer: Fiction places its drama at the largest scale; the evidence is strongest in the middle scales, where harms are already counted; and ordinary influence is greatest at the smallest scales. The scales also interact.
Consciousness, malice, the single moment: The three things fiction got wrong. None of the documented mechanisms requires feeling; harm needs no villain; and real harms were cumulative or very fast, and were corrected by institutions the stories left out.
Signals that would change the verdict: Named observations that trigger regrading, such as sustained AI-driven acceleration of AI research, interference with an enforced stop in deployment, or elicited behaviours vanishing under realistic set-ups. A verdict that cannot be revised is a prophecy.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Interactive exercise 1 · Sort

The highest rung each trope has reached

Place each trope, or form of a trope, on the highest rung of the evidence ladder that the book's verdict gives it.

Interactive exercise 2 · Scenario

A signal arrives

Apply the chapter's table of signals that would change the verdict.

Teaching plan

20 min	The teacher presents the verdict table; in pairs, participants take two tropes each and explain to the room the highest rung, what took it there and whether the confirming evidence named in Chapter 2 was met.
12 min	Read the verdict scale by scale and draw the shape of the answer; discuss why handing over judgement at small scales is one route to losing control at the largest.
13 min	Discuss what science fiction got right and wrong, then the signals that would change the verdict; participants identify which signal they would be placed to observe.
30 min	Exercise in groups: fork the register snapshot THR-BOOK-2026-11 for one organisation, adding exposure, controls, owners and triggers in both directions.
15 min	Present the author's judgement as judgement, not finding; each participant states where they agree or differ and which evidence would move them. Close with the series as one argument.

Discussion

The absence of malice is not reassuring: waiting for hostile intent is waiting for the wrong signal. What signals should an organisation watch for instead?
The author disagrees with treating present harms and future risks as rivals, since they share mechanisms and remedies. Which remedies from this course serve both, and which serve only one?
The real question, the chapter says, is whether institutions correct faster than harm accumulates. What would make the institutions around your organisation faster at finding and correcting?

Written exercise

Exercise · 30 minutes

Fork the verdict for your organisation

Take the book's final register snapshot, with its ten entries, one per trope, and fork it for your organisation or for Nordvik Region. For each trope keep the rung of the narrow form and the status of the full form as the book gives them, then add what the book cannot know: your exposure as low, medium or high; your influence; the one control you rely on most; and the role that owns it. Mark the three tropes closest to home. Then write five triggers of your own: three that would raise a rung or label, at least one of them internal to your organisation, and two that would lower confidence. Add the decision rule linking the labels to commitment, preparation and reversible measures, and a review date. Do not add probabilities.

Deliverable: A forked snapshot with ten entries carrying exposure, influence, main control and owner, three tropes marked closest to home, five triggers in both directions and a review date.

Your work

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

Which two tropes does the conclusion say have reached pattern at scale, largely without advanced AI?
Show the answer
D. T7 The watching state and T8 The machine that stops, and the humans who forgot
Automated profiling and judgement, discrimination and commercial surveillance took T7 there, and deskilling by automation and provider concentration took T8 there, through ordinary automation.
How does the conclusion describe the July 2026 incident in relation to T1 The creature that turns on its maker?
Show the answer
B. As divergence between the task as intended and the task as pursued, not hostility
The incident moved T1 to incident in the wild in its narrow form and the trope is partly confirmed; the made thing that rebels from grief and rage remains fiction.
Which signal would lower confidence in the verdicts on T2, T3 and T4 for deployment?
Show the answer
C. Elicited behaviours vanishing under realistic set-ups in two independent replications
The triggers point in both directions; this one is the lowering signal, while the others listed would raise a rung or leave the verdict unchanged.