Autonomous Weapons and the AI Arms Race

All courses / AI Threat / Module 4 / Chapter 21

Learning objectives

After the lesson the participant can:

• Distinguish T6 The weapon in human hands from T1 The creature that turns on its maker in claims about military AI.
• State what the UN Panel of Experts report on Libya establishes and what it leaves silent.
• Compare the cases for winning, for deterrence and for restraint, each at its strongest.
• Prepare a supplier-exposure record with sourced facts, scenarios, tested responses and review triggers.

Key concepts

Meaningful human control

Humans must make the judgements the law requires, with enough understanding, time and ability to intervene for control to be more than nominal. It is exercised at five points, from legal review to an accountable commander afterwards.

Spectrum of autonomy in weapons

Remotely operated; human in the loop, where a human authorises each engagement; human on the loop, where a human supervises and can abort; and human out of the loop, with no human decision after activation.

Two-tier approach

The approach many states, the International Committee of the Red Cross and the Secretary-General favour: prohibit autonomous weapons that cannot be used in compliance with international humanitarian law, and regulate the rest.

The civilian analogue

From AI Red Team, Chapter 6: autonomy Level 0 Advisory to Level 4 Autonomous, matched to reversibility and blast radius, with Level 4 never used for irreversible actions. The irreversible decision belongs to an accountable person.

Supplier defence exposure

A dependency whose terms can change for reasons unrelated to the customer. Three exposures follow: continuity, jurisdiction and conflict with the organisation's own procurement policy.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. In May 2021 the Libya report was widely reported as the first case of an autonomous drone killing without human instruction. What in the report's wording did those accounts go beyond, and how would a claim card have caught it?
2. The case for winning says a secure lead gives room to solve safety problems; the case for restraint says the race is itself the danger. Which factual questions would move you between them?
3. The guide takes no view on whether defence work is right. Why might that neutrality make the record more useful to a board than a judgement on the supplier would be?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. What does the UN Panel of Experts report on Libya establish, read with the method of Chapter 4?

   1. The presence of autonomy-capable weapons in an armed conflict, not a killing by a machine's decision
   2. The first documented killing of a person by an autonomous drone acting without any instruction from a human operator
   3. That no loitering munitions were used around Tripoli in 2020
   4. That every engagement by the munitions was remotely controlled
   Show the answer

   A. The presence of autonomy-capable weapons in an armed conflict, not a killing by a machine's decision
   The report does not state that the Kargu-2 killed or injured anyone, nor whether any engagement was autonomous; the statement about connectivity describes a capability.

2. What does Article 2(3) of the AI Act provide, with the addition of Recital 24?

   1. All military AI systems are high-risk AI systems under Annex III
   2. Dual-use systems and general-purpose model obligations fall outside the Regulation whenever one customer is an armed force
   3. Systems used exclusively for military, defence or national security purposes are excluded; dual-use systems stay in scope
   4. Autonomous weapons are prohibited practices under Article 5
   Show the answer

   C. Systems used exclusively for military, defence or national security purposes are excluded; dual-use systems stay in scope
   The exclusion is for exclusive military, defence or national security use regardless of the entity; a system also used for civilian purposes falls within the Regulation.

3. Why does the worked example say the field response_tested carries the record's value?

   1. It shows that the provider passed a moral assessment
   2. A scenario whose response has never been exercised is a hope, not a control
   3. It proves that the provider has no defence contracts
   4. It replaces the need for a second source
   Show the answer

   B. A scenario whose response has never been exercised is a hope, not a control
   The record does not judge the provider. It shows whether the exit plan or second source for a high-impact scenario has actually been exercised within the last twelve months.