The Automated State: When Public Authorities Got It Wrong

All courses / AI Threat / Module 3 / Chapter 11

Learning objectives

After the lesson the participant can:

• Describe the three cases and state, for each, what the technology did and what stopped it.
• Identify the four common mechanisms in a description of an automated decision system.
• Explain which AI Act duties answer each mechanism and why a fixed calculation such as income averaging may fall outside the Act.
• Run a harm-path review that names a control with evidence, or an owned gap, for every path.

Key concepts

The harm path

The route from data through a model or rule, a flag, an official action and a burden on the citizen to redress or its absence. Complaints, appeals and inquiries feed back to the model only if someone listens.

Reversed burden of proof

The system's output becomes a presumption the citizen has to rebut, although administrative law in most European states places the burden of establishing the facts on the authority.

Institutional denial

The signals were there, but the institutions treated them as noise. It is the institutional form of T8: officials stopped asking whether the system was right, in organisations that had removed the people whose job was to ask.

Reasons log

A record for every case of why it was handled as it was. With the Article 86 explanation it answers opacity, and it marks the line between decision support and automated decision.

Stop decision

A tested procedure by which named people can stop the system, with a fallback that can run for at least a month. In Nordvik's review the queue reverts to date order.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. In every case the correction came from outside the institution. What does that imply about which outside signals your organisation should treat as monitoring data, and who should receive them?
2. The case for calm says Robodebt's arithmetic would have been harmless had a mismatch prompted a request for evidence. Is the distinction between a harmful tool and a harmful use of a tool useful to an authority deciding what to automate?
3. Nordvik's review leaves one path open and says so. Why might an honest open gap be worth more to an audit committee than a review in which every path is marked controlled?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. On what ground did the District Court of The Hague declare the SyRI legislation unlawful in February 2020?

   1. It breached the AI Act's rules for high-risk AI systems
   2. It did not meet Article 8(2) of the European Convention on Human Rights, the right to respect for private life
   3. It used income averaging to infer debts that did not exist
   4. It had already caused unprecedented injustice to tens of thousands of families, as a parliamentary inquiry had found
   Show the answer

   B. It did not meet Article 8(2) of the European Convention on Human Rights, the right to respect for private life
   The model and its indicators were not public, so neither the people affected nor the court could verify how it worked, and the legislation lacked safeguards against discriminatory and stigmatising effects.

2. Why would Robodebt's income averaging probably not qualify as an AI system under the AI Act, according to the chapter?

   1. It was operated by a public body outside the Union
   2. It affected too few people to be high-risk
   3. It was used only as decision support by officers who examined each discrepancy before a debt was raised
   4. The Act defines an AI system by its capacity to infer; averaging is a fixed calculation
   Show the answer

   D. The Act defines an AI system by its capacity to infer; averaging is a fixed calculation
   A fixed calculation falls outside that definition; it would be caught instead by Article 22 of the GDPR and national administrative law, which is why older protections remain essential.

3. Which path did Nordvik's harm-path review HPR-NR-2026-01 leave open?

   1. Institutional denial: complaints and appeal reversals are not connected to the system
   2. Scale: proxy features raise scores for residents of some areas
   3. Opacity: an appellant cannot learn the system's role
   4. Reversed burden: an applicant must disprove wrong register data
   Show the answer

   A. Institutional denial: complaints and appeal reversals are not connected to the system
   HP-5 is the honest gap, the missing connection to outside signals that let the Dutch and Australian failures run for years; it has an owner and a due date.