Does It Work? Productivity, Quality and the Evidence for Algorithmic Management

All courses / AI Boss / Module 3 / Chapter 15

Learning objectives

After the lesson the participant can:

Classify a study by which of the six outcomes it measured, over what horizon and from what kind of source.
Summarise what the controlled studies of AI assistance and of monitoring found, including their limits.
Explain why gains measured in the first quarter are the least informative evidence about long-run performance.
Design a multi-year evaluation with six outcome categories, a comparison group, stop rules and team-level reporting.

Key concepts

Six outcomes: Output, quality measured independently of the output metric, safety and health, retention, learning, and fairness and trust. The chapter asks of every study which of the six it measured.
Horizon and source: Output effects show in weeks, retention in a year and the pipeline of future managers in several; and a measured outcome, a randomised comparison and a survey perception answer different questions.
Ironies of automation: Bainbridge's observation that automation leaves people the hardest tasks while removing the routine practice that kept them able to do them. For M4 Development it applies to workers and to team leaders.
Staggered roll-out: Introducing a system in waves assigned by a rule fixed in advance, by default at least three waves drawn by lot, so that later waves serve as the comparison group.
Stop rule: A threshold written down before results are known, such as a sustained rise of more than 10 per cent over two quarters that pauses the next wave. A stop rule never triggers a decision about an individual.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Interactive exercise 1 · Sort

The verdict on Q4, label by label

Sort each finding of Chapter 15's verdict under the label the chapter gives it.

Interactive exercise 2 · Put in order

Designing a multi-year evaluation

Put the steps of the chapter's practical guide in order, from the first thing written down to the final decision.

Teaching plan

15 min	The teacher introduces the six outcomes, horizon and source, then hands out one-line summaries of the chapter's studies for participants to sort.
15 min	Plenary on what the sorting shows: gains largest for novices on well-defined tasks, the developer study in which tasks took longer, and the null effects on earnings and hours in Denmark.
20 min	Staged debate in four steps, then the divided verdict on Q4 per function with its labels.
30 min	Exercise: groups design a 24-month evaluation plan for one system and defend their rejected primary outcome.
10 min	Debrief on why both organisations changed their first proposed measure, and why Hansa's matched teams are a weaker design than a staggered roll-out.

Discussion

In the developer study tasks took 19 per cent longer with AI tools, although the developers afterwards estimated a 20 per cent reduction. What does that imply for evidence that rests on what managers and workers report?
The case for concern says algorithmic management may be reporting gains while consuming the capital that produced them: skill, trust and health. What would you need to measure to know?
Employers who consulted workers or their representatives were more likely to report positive effects, yet fewer than half did so. Why might that be, and what would change it?

Written exercise

Exercise · 30 minutes

A 24-month evaluation plan

Write an evaluation plan for one system that performs M1 Direction, M2 Evaluation or M4 Development, using EVP-NR-ROS-2027-01 as the model. State the decision the evaluation informs and who takes it. Name the metric the system itself optimises and record it as the rejected primary outcome, with the reason. Name at least one outcome in each of the six categories, with quality measured independently of the optimised metric. Choose the comparison before roll-out, set the baseline and a horizon of at least 24 months, and write one stop rule with threshold and action. Fix the reporting level, the minimum group size, the prohibition on consequence use, the independent analyst and the parties who agree the plan.

Deliverable: A versioned evaluation plan with rejected primary outcome, six outcome categories, comparison design, horizon, stop rule, reporting rules, analyst and agreeing parties.

Your work

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

Which measure did Nordvik Region record as the rejected primary outcome in EVP-NR-ROS-2027-01, and why?
Show the answer
B. Visits per paid hour in the first quarter, because the system itself optimises it and it says nothing about quality, health or retention
The supplier's proposed success measure was kept as one output measure but rejected as primary: the representatives objected that it would rise as soon as routes were tightened, whatever happened to care or staff.
What did the meta-analysis of 63 studies of electronic monitoring, as Eurofound's review reports it, find?
Show the answer
D. No average gain in performance, together with lower job satisfaction and higher stress
The chapter contrasts it with the restaurant study: monitoring that targets a specific misconduct can pay, but as a general management method it does not, on average, raise performance.
What is the chapter's verdict on open question Q4, whether algorithmic management works over years?
Show the answer
A. It remains open: short-run gains and metric gaming are observable, long-run effects are expected to be mixed, and large organisation-wide gains are speculation
The verdict is divided, and the chapter adds that most organisations do not know whether it works because they have not designed a way to find out; the remedy is a multi-year evaluation agreed in advance.