Trustworthy AI: Transparency, Fairness and Human Oversight

All courses / AI Governance / Module 3 / Chapter 10

Learning objectives

After the lesson the participant can:

• Distinguish the three audiences of transparency and what each needs.
• Explain why independence, separation and sufficiency cannot all hold when base rates differ, and where the resulting choice is recorded.
• Design human oversight that addresses workload, defaults, automation bias and the stop function.
• Outline a fundamental rights impact assessment in which every risk has an identifier, a measure and a trigger.

Key concepts

System-level transparency

For large language models, an explanation of the model's own output is generated text, not a faithful account of the computation. The honest alternative records which sources were retrieved, which tools were called, which model version answered and which verification steps applied.

Three families of fairness criteria

Independence (demographic parity), separation (equal error rates across groups) and sufficiency (calibration). They cannot all hold when base rates differ, so every system embodies a choice, made explicitly and recorded in the AI Decision Record.

Oversight patterns

Human-in-the-loop: a person approves each output before it takes effect. Human-on-the-loop: the system acts and a person monitors and can halt. Human-in-command: a person decides whether and where the system is used at all.

Automation bias controls

Measures that show whether oversight has become ceremonial: a blind sample of cases shown to the overseer without the score, and both a lower and an upper override-rate alert, because ceremonial oversight and a failing model both show in that number.

Fundamental rights impact assessment

The assessment required of certain deployers by Article 27, covering processes, period and frequency of use, affected groups, specific risks, oversight measures and measures if risks materialise. Kept in the repository with stable section identifiers so that other artefacts can cite individual findings.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. An overseer who must process three hundred recommendations an hour with a single accept button has no control, whatever the policy says. Where in your organisation does the workload decide the quality of oversight?
2. Nordvik Region's notice leads with what the system does not do and offers handling without the system's suggestion. What does that option demand of the workflow behind it?
3. The FRIA names groups such as paper applicants that a purely statistical fairness analysis would not have found. How would you find the equivalent groups for a system in your own service?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. Why does Nordvik Region's oversight design send five per cent of cases to the overseer without the score?

   1. To reduce the overseer's workload during peak periods
   2. To give a baseline for measuring agreement with the system, as a check on automation bias
   3. To comply with the minimum group size of 200 that the fairness configuration sets for each subgroup
   4. To test whether the stop function reaches a safe state in time
   Show the answer

   B. To give a baseline for measuring agreement with the system, as a check on automation bias
   The blind sample gives a baseline of the overseer's own judgement. Agreement with the system far above that baseline signals automation bias, which Article 14 asks the design to address.

2. Where, according to the chapter, should protected attributes be joined to predictions for fairness measurement?

   1. In the production feature set, so that monitoring is continuous
   2. In the caseworker's interface, so that the overseer can see them
   3. Nowhere, since special categories may not be processed for this purpose
   4. Only in an isolated evaluation environment, with the join logged
   Show the answer

   D. Only in an isolated evaluation environment, with the join logged
   Article 4a of the AI Act permits such processing for bias detection where strictly necessary and under safeguards. The chapter's rule is to join protected attributes only in an isolated evaluation environment, never in the production feature set, and to log the join.

3. In the benefits-triage oversight design, what is the effect of triggering the stop function?

   1. The queue reverts to date order
   2. All pending applications are approved automatically
   3. The supplier rolls the model back to its previous version
   4. Rankings continue but are hidden from caseworkers
   Show the answer

   A. The queue reverts to date order
   The stop function reverts the queue to date order, the safe state the service ran on before the system existed, within 60 seconds, and it is tested quarterly.