Gradual Disempowerment and Correlated Failure

All courses / AI Threat / Module 5 / Chapter 26

Learning objectives

After the lesson the participant can:

• Explain the gradual disempowerment argument and why it requires neither misaligned AI nor a sudden event.
• Distinguish decisive from accumulative existential risk and the evidence each calls for.
• Describe how correlated failure arises from interactive complexity, tight coupling and shared providers, and what kind of control is fast enough.
• Write a delegation register entry and design a reversibility test with criteria fixed in advance.

Key concepts

Gradual disempowerment

The argument that systems stay responsive to people partly because they depend on human participation. Each time AI replaces human labour or cognition, that dependence and the reason to remain responsive weaken, without anyone intending it.

Decisive and accumulative risk

Kasirzadeh's vocabulary: the decisive type is an abrupt catastrophe; the accumulative type arises from many smaller disruptions that build up and weaken resilience until a shock that would once have been absorbed causes collapse.

Circuit breaker

A pre-decided limit that acts at machine speed without waiting for a person, such as budgets, rate limits and automatic pauses. It is often the only form of human control fast enough, as the flash crash showed.

Delegation register

A list of every material function handed to an AI system, with the decision behind each step, the usage mode, the human role that remains and the path back. It lists functions, not systems, because what matters is the human capacity no longer exercised.

Reversibility test

A test of whether the organisation could perform a delegated function at an acceptable level without the AI system for a defined period. A failed test freezes further delegation in that function until the path back is restored.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. Risk registers record fallback to manual process as a mitigation without anyone having run the manual process for years. Which fallback in your organisation would you least like to be asked to demonstrate tomorrow?
2. Hansa's test found that an evaluation check used to approve the fallback called the primary provider's model. Why are hidden dependencies of this kind so hard to see in normal operation?
3. The concern side holds that the ability to change course erodes as delegation deepens; the calm side holds that societies and organisations can change course. What would count as evidence for each in your own setting?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. Why does the gradual disempowerment argument not require misaligned AI, according to the chapter?

   1. It assumes that AI systems will be switched off in time
   2. It concerns only the individual mind and not institutions
   3. Systems that do exactly what their operators ask, in competition, are enough
   4. It depends on a single system gaining a decisive strategic advantage over its rivals
   Show the answer

   C. Systems that do exactly what their operators ask, in competition, are enough
   The implicit check is that states, firms and culture need people. As AI replaces participation, that need and the responsiveness it produced weaken, without any system having goals of its own.

2. What does the 2010 flash crash show about remedies for correlated failure at machine speed?

   1. Human operators can always intervene in time if they are alert, as the traders did during the afternoon of 6 May
   2. Removing all algorithms from markets was the only solution
   3. Faster algorithms correct the errors of slower ones
   4. Automatic pauses and circuit breakers designed by people in advance act fast enough where people cannot
   Show the answer

   D. Automatic pauses and circuit breakers designed by people in advance act fast enough where people cannot
   No algorithm malfunctioned; a proxy failed. A brief automatic pause helped the market recover, which is why pre-decided limits belong in any agent-run process.

3. What was the result of Nordvik's reversibility test RVT-NR-2027-01, and what followed from it?

   1. Passed with conditions; further delegation of benefits triage is frozen until the conditions are met
   2. Failed; the claims ranker was permanently withdrawn and the intake queue reverted to date order for good
   3. Passed; delegation was extended to automatic decisions
   4. Not run; the test was replaced by a desk review
   Show the answer

   A. Passed with conditions; further delegation of benefits triage is frozen until the conditions are met
   The checklist referred to a retired form, two newer caseworkers had never triaged unranked claims and capacity was short; each finding is cheap to fix now and costly in a real outage.