Measuring the Loop: Cycle Time, Yield, Re-test Pass Rate and the Metrics That Mislead

All courses / AI Purple / Module 5 / Chapter 18

Learning objectives

After the lesson the participant can:

• Compare the six recommended measures with the four that reward activity, giving the replacement for each.
• Explain the four Goodhart mechanisms with an example from the loop for each.
• Draft measure catalogue entries with definition, denominator, grain and gaming check.
• Design a board page of four to six measures that respects the aggregation rules agreed with employee representatives.

Key concepts

Six measures of the loop

Finding-to-control cycle time, finding half-life, re-test pass rate at first attempt with held-out variants, verified share within expiry, yield per exercise and detection debt. Each has a denominator and cannot be improved by running more scenarios.

Measures that mislead

Scenarios run, findings opened, findings closed and a green percentage. They are useful as operational data on cost and load but reward activity when they become targets or headline figures.

Goodhart mechanisms

Regressional, extremal, causal and adversarial, after Manheim and Garrabrant. AI sharpens the adversarial case: a drafter rewarded for passing scenarios produces detections that pass scenarios.

Yield per exercise

Findings per exercise that no catalogued run could have produced, and how many reach verified control within 90 days. It is the only one of the six that measures what the people on the red side add.

Grain and minimum group size

Every measure is computed per technique, scenario, class, service or team, never per named person. A role held by one person is that person. Hansa Mutual's floor is five people; Nordvik Region reports the unit of six as a whole.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. Nordvik Region's first closure after 28 days would have been the headline; the verified closure took 112 days. Which of your own headline figures would change if only verified closure counted?
2. The chapter states the price of aggregation openly. How should a manager find out who needs support without turning the loop's records into a ranking?
3. Step nine of the guide doubles the scenarios run in a test quarter to see whether any figure moves. What would that test do to your present dashboard?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. Which measure does the chapter recommend in place of findings closed?

   1. Scenarios run per quarter, with the share above S1.
   2. The green percentage over the whole scenario library.
   3. Finding half-life, with the recurrence rate beside it.
   4. Findings opened per exercise by the red side.
   Show the answer

   C. Finding half-life, with the recurrence rate beside it.
   Findings closed can be gamed by closing on merge, closing by acceptance or reclassifying a repeat as new. The chapter reports finding half-life, which counts only verified closure, and recurrence instead.

2. At what grain may a measure of the loop be computed under the chapter's rules?

   1. Per technique, scenario, class, service or team, never per named person.
   2. Per analyst, provided the figures are shown only to the CISO.
   3. Per shift, so that out-of-hours performance can be compared.
   4. Per role holder, including roles that are held by one person.
   Show the answer

   A. Per technique, scenario, class, service or team, never per named person.
   No measure is computed per named person, and a role held by one person is that person, so by-owner and by-shift figures are excluded. The catalogue lists person, role held by one person and shift as forbidden grain.

3. Which Goodhart mechanism is at work when a detection drafter rewarded for passing scenarios produces detections that pass scenarios?

   1. Regressional.
   2. Extremal.
   3. Causal.
   4. Adversarial.
   Show the answer

   D. Adversarial.
   The adversarial mechanism concerns agents that respond to the measure, people and, in this book, software. The chapter says AI makes this form faster, which is why part of the instrument is hidden and a person reproduces a sample.