Weighing the Evidence and Disagreeing Well

All courses / AI Threat / Module 1 / Chapter 4

Learning objectives

After the lesson the participant can:

Grade a claim at the highest rung its evidence reaches and record the quality of that evidence.
Apply the five questions for a laboratory result to separate capability from propensity.
Explain why multiplying likelihood by impact fails for improbable, extreme and contested risks and what the book uses instead.
Design a constructive discussion that ends in no-regret actions, owned cruxes and stated value questions.

Key concepts

Three rules of the ladder: A claim is graded at the highest rung its evidence reaches, with that evidence's quality; it must be stated precisely first; and absence of evidence is graded too, because sought and not found differs from never looked for.
Contrived test: A scenario built to elicit a behaviour. It shows that under its conditions the system produced the behaviour at some rate; contrivance lowers the evidence for propensity, not for capability.
Four attributes for tail risks: The book holds tail risks with rung and label, consequence, reversibility, and the extent of expert disagreement, and treats them through scenario analysis, reverse stress tests and leading indicators with triggers.
Crux: A factual question whose answer would change a side's position. Naming cruxes, each with an owner and a trigger, turns a debate into a research agenda.
No-regret action: An action worth taking whatever the cruxes show, such as pre-deployment testing, a stop path independent of the system's cooperation, skills to work without AI, incident sharing and reversible dependence on providers.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Interactive exercise 1 · Put in order

The evidence ladder

Put the five rungs in order, from the lowest rung to the highest.

Interactive exercise 2 · Sort

Labels and the decisions they justify

The labels grade the outlook and tie it to a kind of decision. Sort each description under the label it belongs to.

Teaching plan

15 min	The teacher presents the five rungs with the chapter's examples and the three rules; participants rewrite the ungradable claim that AI resists shutdown into a precise one.
12 min	Work through capability, propensity and harm and the five questions for a contrived test, using the blackmail and in-context scheming experiments as described by set-up and findings only.
13 min	Explain labels and decisions, tail risks and the reverse stress test, then the five biases; each participant names the error their organisation is most prone to.
35 min	Exercise in groups of three, with grader, challenger and register owner: grade one claim into an evidence record, run the challenge one rung up and one down, and record dissent.
15 min	Read Nordvik's crux map CRX-NR-2026-01 and discuss how the most expensive option left the table without anyone conceding the larger argument.

Discussion

A behaviour demonstrated in fifty laboratories remains a laboratory demonstration. What would have to be observed, and where, for it to move up, and who in your sector would be placed to observe it?
Two council members agree on the facts about a staff assistant and still disagree. How should a governing body handle a disagreement that no evidence can settle?
The chapter says the symptom of doom is that no observation could lower the estimate. What is the matching symptom of normalcy bias, and how would you test your own team for both?

Written exercise

Exercise · 35 minutes

Grade a claim into an evidence record

Take one claim about AI threat from a briefing, a news report or a vendor presentation. Following the eight steps of the practical guide, state it precisely with system class, conditions and behaviour and link it to a trope; list the primary sources in one line each; write what is established for capability, propensity and realised harm, using not established where the sources are silent; assign the rung and quality, answering the five questions if it is a laboratory result; record the absence of evidence; assign the label with a one-sentence reason and name one observation that would raise the rung. The challenger then argues one rung up and one down. Record any remaining dissent and set a review date.

Deliverable: A completed evidence record with claim, distinctions, rung, quality, label, interpretation of at most three sentences, dissent and review date.

Your work

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

The chapter says dismissing a laboratory result because its scenario was contrived is an error. Why?
Show the answer
D. Contrivance lowers the evidence for propensity, not for capability
The capability is shown regardless of how the scenario was built, and July 2026 showed that circumstances nobody designed can trigger it. What contrivance weakens is the inference to propensity.
What does a reverse stress test allow an organisation to do with a risk such as systemic loss of control?
Show the answer
B. Start from an outcome it could not survive and ask what would produce it
The reverse stress test starts from an unsurvivable outcome and works backwards to the chain of events and indicators, which avoids choosing between camps on a probability.
In the graded evidence record EVD-024-01, how is the disagreement between grader and challenger handled?
Show the answer
A. The dissent is recorded in the record, not resolved by seniority
The challenger argued for expected instead of observable; the dissent stays in the record because the purpose is to make the argument reviewable, not to win it.