Fairness, Bias and the Appeal That Never Comes

All courses / AI Boss / Module 3 / Chapter 12

Learning objectives

After the lesson the participant can:

• Explain the four mechanisms by which bias enters a management system without anyone intending it.
• Identify the four ways in which a route of appeal fails.
• Compare the case for concern and the case for calm on fairness, and state the four conditions on which they meet.
• Design a disparate-impact test with representatives and interpret a triggered result.

Key concepts

Four mechanisms of bias

The definition of good performance, proxies that correlate with protected characteristics, historical data that codifies earlier decisions, and feedback loops in which outputs influence future inputs.

Indirect discrimination

An apparently neutral provision that puts persons with a protected characteristic at a particular disadvantage unless objectively justified by appropriate and necessary means, with the burden of proof shifting to the respondent once a presumption arises.

The appeal that never comes

Contest fails in four ways: no notice that a score exists, no reasons beyond a threshold, no one to ask with authority to change the decision, and a remedy that reverses one case while the rule stays.

Disparate-impact test

A measurement of each group's rate of a favourable outcome relative to a reference group, with a reporting floor of 20 per cell. A lower bound below 0.80 triggers investigation and is not a finding of discrimination.

Correcting the system, not only the case

A successful contest goes to the owner of the system so that the rule is changed and the affected cases are listed for their deciders; a correction that leaves the rule in place corrects nothing.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. The Bologna score rewarded reliability and had no field for a reason. What other apparently neutral definitions of good performance could penalise the exercise of a right?
2. The case for calm says a rule in software can be run on every case and compared by group, whereas the judgement of three hundred line managers cannot. How far does auditability answer the case for concern?
3. Hansa Mutual flagged promotion files for review after finding the disparity. Who should review them, and what should the people concerned be told?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. What caused the triggered result for part-time handlers in DIT-HM-CTA-01?

   1. Team leaders used their right to reassign under DLG-HM-CTA-01 to move complex claims away from part-time handlers by hand
   2. The agent used sex as a direct input to the allocation of complex claims
   3. Part-time handlers had stated a preference for routine claims under PREF-01
   4. Rule CTA-R-14 offered complex claims only to handlers with two consecutive working days left in the week
   Show the answer

   D. Rule CTA-R-14 offered complex claims only to handlers with two consecutive working days left in the week
   A rule written for continuity of claims handling put part-time handlers, most of them women in this population, at a particular disadvantage in access to the work that leads to promotion, without a justification that would pass the test.

2. How does the chapter's practical guide treat a ratio whose 95 per cent interval has a lower bound below 0.80?

   1. As a finding of unlawful discrimination that must be reported to the authority
   2. As a trigger for investigation, not a finding of discrimination
   3. As within tolerance provided the reference group is larger
   4. As grounds for dismissing the manager who owns the system
   Show the answer

   B. As a trigger for investigation, not a finding of discrimination
   The trigger starts a search for the cause within 20 working days and a recorded decision on whether the rule is objectively justified. The measurement alone settles nothing about justification.

3. What does the study of more than a million rideshare drivers show, according to the chapter?

   1. A rule treating every individual identically can produce an outcome gap between groups
   2. Algorithmic pay formulas are proven to be fair to all groups
   3. Customer discrimination explains most of the gender earnings gap
   4. Pay algorithms always contain a hidden variable for the driver's sex
   Show the answer

   A. A rule treating every individual identically can produce an outcome gap between groups
   A gap of roughly 7 per cent arose under a formula that does not know the driver's sex, explained by experience, preferences over where to work and driving speed. Whether such a gap is acceptable is a question for those who set the rule.