Measuring and Leading AI Adoption in Development

All courses / AI Developer / Module 4 / Chapter 20

Learning objectives

After the lesson the participant can:

Classify measures into the four families of use, experience, delivery and outcome, and give each a gaming check
Explain why acceptance rate and share of generated lines cannot carry a value claim
Design an internal experiment with a comparison, a counting rule, an independent oracle and a stop rule fixed in advance
Draft an adoption and value record that files the quoted and the measured figure side by side

Key concepts

Portfolio of use cases: An adoption programme is a portfolio of use cases by work class and usage mode, each with a sponsor, a hypothesis and a stop rule, and an evidence owner who is not the sponsor. A seat is an input, not an outcome.
Four families of measure: Use (reach and willingness to try), experience (how the work feels), delivery (whether one service got faster or less stable) and outcome (whether what shipped stayed shipped). Each measure carries a gaming check: a second measure that does not use the same data.
Acceptance rate: The share of shown suggestions a developer accepts. It measures a reaction to a suggestion and not the fate of the code, and its level depends on the configuration, so it is valid for one configuration and date.
Stop rule: A programme without one has the structure of a ratchet. A stop rule has a condition tied to evidence and not to mood, a stop path outside the tool and the sponsor, and a way to stop part of a use case.
Adoption and value record: The artefact AVR: a short versioned file in which the quoted claim and its basis are written down beside the measured figure, with what is pending and an evidence age. Claims are shown as verified, expired or asserted.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Interactive exercise 1 · Sort

Four families of measure

Sort each measure into the family the chapter's table places it in: use, experience, delivery or outcome.

Interactive exercise 2 · Scenario

The stop condition is met and the sponsor disagrees

Apply the chapter's three requirements for stopping a use case that does not work.

Teaching plan

15 min	Lecture on adoption as a portfolio and on the four families of measure; participants sort the figures their organisation reports into the families.
20 min	Staged debate on whether AI is making developers more productive: the case for concern, the case for calm, the table where the positions meet, and the verdict.
15 min	The teacher presents the three experiment designs and what the published trials teach about selection, refusals and the smallest effect a small team can resolve.
30 min	Exercise: groups design an internal experiment and the record that will hold its result.
10 min	Groups read out their stop rule and name who holds the licence, the gateway route and the repository permissions.

Discussion

With 12 pilot developers and six sprints, Nordvik Region can show a large effect and cannot show a small one. How should a small organisation decide when it cannot resolve the effect it cares about?
Both sides of the debate accept that throughput and stability are read together. Which of your own figures is reported without its counterpart?
The counting rule is agreed with the representatives before any data are collected. What would change if it were agreed afterwards?

Written exercise

Exercise · 30 minutes

Design an internal experiment and its record

Take an adoption claim in your organisation, or Nordvik Region's claim that the pilot made developers a third faster. Write the claim in one sentence, who quotes it and its evidence age. Write the scope: work class, usage mode, service and period. Choose a design, for example a task-level comparison with a recorded coin flip at planning, and state what it cannot control for and the smallest effect it can resolve. State the counting rule: no figure for a named developer and a minimum group size. Fix outcomes in the four families, each with a gaming check and a reading date. Say how the oracle stays independent. Write the stop rule in advance.

Deliverable: An experiment plan and an adoption and value record skeleton with quoted claim, scope, design, counting rule, outcomes, stop rule and expiry.

Your work

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

What does the chapter mean by a gaming check?
Show the answer
A. A second measure, from different data, that would move if the first were gamed
Each measure in the four families carries a gaming check, a second measure drawn from different data that would move if the first were being gamed, because Goodhart's law applies when a measure becomes a target.
Who should own the evidence for a use case in the adoption portfolio?
Show the answer
C. An evidence owner who is not the sponsor of the use case
Each use case has a sponsor, a hypothesis and a stop rule, and an evidence owner who is not the sponsor, so that the person who wants the use case to continue does not also judge whether it works.
What state are the two claims in the Nordvik record AVR-NR-2030-01 in at the start of the year, with an evidence age of five months?
Show the answer
D. The usage claim has expired and the value claim is asserted
With the 90-day default for usage, a five-month-old usage figure has expired. The value claim of a third faster was never measured against delivery, so the record shows it as asserted.