What Going Well Looks Like

All courses / AI Threat / Module 7 / Chapter 34

Learning objectives

After the lesson the participant can:

• State the optimistic case at full strength, including the cost of not building, and the three qualifications that belong with it.
• Explain why compressed progress and broadly shared progress are different claims.
• Describe the five conditions for going well and link each to measures from the threat chapters.
• Write a positive-scenario register entry with outcome, conditions, indicators and the threat that grows if a condition fails.

Key concepts

The cost of not building

A decision not to use a technology also has victims, but they are statistical and invisible: the patient diagnosed later, the applicant who waited longer. A risk function that records only the downside tends to recommend inaction, which is seldom recorded as a decision.

Direction of technology

Acemoglu and Johnson's claim that the direction is a choice: a vision of replacing workers often yields so-so automation, while a vision of making people more capable, backed by countervailing powers, spreads the gains.

Same capability, different objective

Fluent, personalised argument is neutral as to its objective. A system optimised for engagement finds flattery and outrage useful; one instructed to be accurate behaves differently. The objective belongs in the decision record.

Five conditions for going well

Capability that is real and measured; safety and containment that do not depend on intent; distribution of gains decided, not assumed; institutions that can see, correct and stop; human capability kept in practice.

Positive-scenario entry

A register entry with a direction field and its own prefix that records a good outcome, the conditions it depends on, an indicator for each and the threat entry that would grow if a condition failed.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. Inaction is seldom recorded as a decision at all. How could your organisation's risk process make the cost of not using AI for a given purpose visible alongside the cost of using it?
2. POS-NR-01 lets the committee ask whether the system should be used at all if an independent evaluation finds no gain. What would count as no gain for an AI system you rely on, and who would be willing to act on that finding?
3. The authors who raise AI welfare warn of two opposite errors. Why does the book take no position, and what small practical implication does it still draw for deployers?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. On which rung of the evidence ladder does the chapter place the most ambitious claims of the vision of compressed progress?

   1. Pattern at scale, because productivity gains are measured
   2. Incident in the wild, because diseases have been cured
   3. Argument, because they follow from premises about future capability not yet demonstrated
   4. Fiction, because the essay is a work of imagination
   Show the answer

   C. Argument, because they follow from premises about future capability not yet demonstrated
   The essay is a well-argued statement by an interested party, not a forecast with evidence behind it, and it treats distribution as a problem to be solved, not an automatic outcome.

2. What did the experiments with an AI mediator in the United Kingdom find?

   1. Participants preferred the AI-drafted group statements in a majority of comparisons and were less divided afterwards
   2. Participants rejected every statement drafted by a model
   3. AI mediation was shown to survive a polarised election campaign in which thousands of participants took opposing sides
   4. Groups became more divided after reading the statements
   Show the answer

   A. Participants preferred the AI-drafted group statements in a majority of comparisons and were less divided afterwards
   The finding is a laboratory demonstration from structured online settings; it shows that the capability feared as manipulation can help people find statements they all endorse, not that it holds in a real campaign.

3. What does the positive-scenario entry show a governing body about the controls in the threat register?

   1. That the controls can be removed once benefits appear
   2. That the hopeful and the defensive case need two separate budgets and two separate owners
   3. That only security controls contribute to good outcomes
   4. That most controls are conditions of getting value, not merely costs of avoiding harm
   Show the answer

   D. That most controls are conditions of getting value, not merely costs of avoiding harm
   Every condition in POS-NR-01 is already covered by a control adopted against a threat, so the hopeful and defensive cases are one programme.