Superintelligence, Loss of Control and the End of the World

All courses / AI Threat / Module 5 / Chapter 27

Learning objectives

After the lesson the participant can:

• Explain the intelligence explosion, the orthogonality thesis, instrumental convergence and the treacherous turn as steps of an argument.
• Grade each of the six premises on the evidence ladder and state where the chain weakens.
• Compare the strongest objections and explain why published probability estimates do not belong in the likelihood column of a register.
• Write a register scenario that uses a reverse stress test, no-regret measures and leading indicators in both directions.

Key concepts

The chain of six premises

Carlsmith's testable form of the argument: advanced, planning, strategically aware systems become feasible; strong incentives to build them; alignment is much harder; some misaligned systems seek power; this scales to permanent disempowerment; that is an existential catastrophe. A reader who rejects the conclusion must say which premise fails.

Orthogonality and instrumental convergence

Almost any level of intelligence could be combined with almost any final goal, so capability says nothing about benevolence; and agents with different final goals would pursue similar intermediate goals such as self-preservation and resources.

Grown, not crafted

The point that training selects for behaviour without revealing the goals behind it, so a system's internal drives may only correlate with the intended ones, and goals cannot yet be verified before a system could resist correction.

Capability versus power

The distinction in the normal-technology objection: power is something humans grant through deployment, permissions and oversight. The objection denies no laboratory finding; it denies that the findings imply the fifth premise.

Reversibility, option value, no-regret

The three principles for a risk whose probability cannot be estimated, whose consequence exceeds any scale and which is irreversible: keep the ability to change course, preserve what is cheap now and impossible later, and take measures justified by existing evidence alone.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. Which of the six premises do you find weakest, and what observation would change your mind about it?
2. The objections show that the argument is not a proof, not that the risk is negligible. How should a governing body speak about a risk in that position without either alarming or dismissing?
3. Nordvik's audit committee concludes that civilisational loss of control lies outside the Region's mandate, yet it still acts. What exactly does it do, and why is that a reasonable answer?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. When the six-premise chain is held to the evidence, what does the chapter find?

   1. Every premise rests on a pattern at scale
   2. Early premises well evidenced, middle ones laboratory demonstrations, final ones argument alone
   3. No premise has any support beyond fiction
   4. The final premises, on permanent disempowerment, are the best evidenced of all six in the chain
   Show the answer

   B. Early premises well evidenced, middle ones laboratory demonstrations, final ones argument alone
   Incentives to build reach pattern at scale and alignment difficulty laboratory demonstration, but scaling to permanent disempowerment has no observation behind it: the components exist, their assembly into catastrophe is not shown.

2. Why does the chapter say published probability estimates of existential risk do not belong in the likelihood column of a risk register?

   1. They express belief under deep uncertainty and do not measure a quantity
   2. They are all below one in a million and can be ignored
   3. They were produced only by developers with commercial interests in the answer
   4. They are derived from frequency data that is out of date
   Show the answer

   A. They express belief under deep uncertainty and do not measure a quantity
   Estimates span from well under one per cent to about one in ten. They rule out treating the tail as fiction, but a single number would record a choice between camps.

3. What decision rule does Hansa's entry THR-HM-09 record?

   1. Halt all use of frontier models across the group
   2. Raise capital by the amount of the estimated loss
   3. Delegate the question to the model providers
   4. Reversible measures only; no change to AI strategy on this entry alone
   Show the answer

   D. Reversible measures only; no change to AI strategy on this entry alone
   The entry is at the rung of argument and labelled speculation, with influence none; every control is a no-regret measure already justified elsewhere, and the chief actuary records that it changes no capital figure.