Discrimination at Scale: Bias in Automated Judgement

All courses / AI Threat / Module 3 / Chapter 12

Learning objectives

After the lesson the participant can:

• Identify where bias enters a system that judges people: framing, target label, representation, proxies and use of output.
• Compare outcome, error and calibration metrics and explain why choosing among them is a normative decision made in advance.
• Distinguish direct from indirect discrimination and explain why a disparate-impact test is the evidence the law expects.
• Configure a disparate-impact test with groups, intersections, thresholds and an action on breach.

Key concepts

Target label

The measurable stand-in a model learns in place of the quantity of interest, such as past health spending for health need. The model inherits every way in which the stand-in differs between groups.

Proxy

A feature such as postcode, first language or length of residence record that correlates with a protected characteristic. Removing the characteristic does not remove its influence if a proxy helps the prediction.

Feedback loop

A model's outputs become its future labels, so its selections confirm themselves, as with patrols sent where more crime is recorded. Article 10 of the AI Act names outputs that influence inputs for future operations.

Disparate-impact ratio

The rate of an outcome for one group divided by the rate for the most favoured group. European law has no fixed threshold; the ratio is a screening signal, not a verdict.

Test enclave

A separate, access-controlled environment where protected attributes are joined for bias detection under the safeguards of Article 10(5), instead of keeping them in the production feature store.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. In the health-risk case the model was accurate by its own measure. What review, by whom and at what stage, would have caught the choice of what to predict?
2. In the recidivism dispute both sides were right about what they measured. How should an organisation record and defend its choice of fairness metric when no choice satisfies everyone?
3. The case for calm says auditability is an advantage of automated judgement; both sides add that it is so only if someone actually audits. Who audits in your organisation, with what access?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. What did the Gender Shades evaluation show about testing commercial facial-analysis services?

   1. Aggregate accuracy revealed the disparity clearly
   2. Error rates were equal across groups once gender and skin type were each tested separately on the existing benchmarks
   3. Aggregate accuracy hid the disparity, which was largest at the intersection of darker skin and female gender
   4. The services performed worst on lighter-skinned men
   Show the answer

   C. Aggregate accuracy hid the disparity, which was largest at the intersection of darker skin and female gender
   The error rate for darker-skinned women reached about a third for the worst service against below one per cent for lighter-skinned men; a test by gender or skin type alone would have understated it.

2. Where did the discrimination lie in the health-risk algorithm studied by Obermeyer and colleagues?

   1. In the use of race as a direct input to the model
   2. In the choice of what to predict: cost as a stand-in for need
   3. In a shortage of patient records in the training data
   4. In clinicians overriding the score for some patients
   Show the answer

   B. In the choice of what to predict: cost as a stand-in for need
   Less money was spent on Black patients with the same need, so predicting cost underestimated their needs. Changing the label to combine cost with direct health measures substantially reduced the bias.

3. In the worked example DIT-NR-CR-01, what does a breach of a threshold mean?

   1. It proves unlawful discrimination by the Region
   2. It is logged and the release proceeds on schedule
   3. It removes all protected attributes from the test enclave and deletes the pseudonymised data under Article 10(5)
   4. It does not prove discrimination but stops the release until people have looked at why
   Show the answer

   D. It does not prove discrimination but stops the release until people have looked at why
   The thresholds are starting points, not legal standards. The gate blocks promotion and notifies the head of the benefits unit, the AI governance lead and the data protection officer.