Case Studies

All courses / AI Governance / Module 5 / Chapter 18

Learning objectives

After the lesson the participant can:

• Analyse a case through the three lenses of usage mode, AI Act role and controls that worked, failed or were missing.
• Explain how decision support was made a structural boundary in the social insurance case.
• Compare what a deployer controls in Mode 4 with what it controls in Modes 1 and 2, using the documented cases.
• Identify the patterns the composite cases share and apply them to a system in the participant's organisation.

Key concepts

Anonymised composite

A case that combines several organisations and engagements with details altered, so that none describes a real, named organisation although the pattern is real. Five of the chapter's eight cases are composites; three are documented and can be checked.

Structural boundary

A limit on what the AI system may do that is built into the design, such as a model producing a priority score and a list of likely missing documents but never a proposed decision, with the decision record carrying no reference to the score.

Registry as the condition of production

The decisive organisational move in the bank case: a model not in the registry was not in production, and inference endpoints could load only approved artefacts. The number of models fell by a fifth as units found models nobody used.

Verifiability

What the SyRI risk-scoring system lacked. The risk model and its indicators were not public and its application was insufficiently transparent and verifiable, so the court held that the legislation did not strike the fair balance required and declared it without binding effect.

Deployer's levers in Modes 1 and 2

In the UK and Italian cases most controls belonged to the provider, leaving the deployer permissions, the service tier and contract, and a fallback for dependent processes, since a service can be withdrawn from one market at short notice.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. In the social insurance case officers could explain their decisions but not the score, and claimants were told it was a scheduling aid. Under what conditions would that answer stop being adequate?
2. The bank forced three business-critical models into an uncomfortable exception process and judged the discomfort to be the point. When is discomfort a sign that governance works, and when a sign that it is wrong?
3. The UK evaluations found a self-reported saving of 26 minutes a day and, with control groups, no evidence that time savings had improved productivity. What does the difference tell you about how to design your own pilots?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. In the manufacturing case, how did the group take its maintenance agent out of the employment-related high-risk category?

   1. It limited the agent to plants outside Germany
   2. It removed task allocation to individuals: the agent proposes a skill profile, a supervisor assigns
   3. It replaced the agent's model with a smaller one hosted in the Union
   4. It asked each technician for written consent to automated scheduling and recorded it in the permission manifest
   Show the answer

   B. It removed task allocation to individuals: the agent proposes a skill profile, a supervisor assigns
   Allocating work to named technicians could fall within the high-risk employment categories and was subject to co-determination. The manifest forbids an assignee and disallows scheduling a technician, and the works council's agreement is recorded in the AI Decision Record.

2. What did the hospital group decline to do, and why?

   1. Run the triage tool in more than two hospitals, because of cost
   2. Report the lower local sensitivity to the vendor, because of the contract
   3. Issue data permits to researchers, because of the data protection officer's objections
   4. Improve the vendor's tool with local data, because that would make it a provider
   Show the answer

   D. Improve the vendor's tool with local data, because that would make it a provider
   Improving the CE-marked tool with local data would make the hospital a provider and bring the work under secondary use. The group has so far declined to cross that line, which the case records as an outcome.

3. What did the municipality's regression suite of golden questions catch within a year?

   1. Two vendor model updates that would have reintroduced eligibility answers
   2. A leak of residents' personal data to the vendor's training set
   3. A fall in the share of first-line enquiries handled by the service
   4. Several conversations in which the disclosure had not been shown
   Show the answer

   A. Two vendor model updates that would have reintroduced eligibility answers
   The city's version-controlled scope excludes eligibility assessments. The golden set runs on every content update and vendor model change, and it caught two model updates that would have brought eligibility answers back.