Measuring Operations: The Four Key Metrics, SLOs, Toil and the Metrics That Mislead

All courses / AI Operations / Module 5 / Chapter 20

Learning objectives

After the lesson the participant can:

State for a given metric the decision it informs, who takes it and what the decider does differently
Explain three limits of the four key metrics as a measure of operations
Replace a mean time to restore with a distribution, a count beyond the objective and a named list
Draft a metric definition with clocks, source, form, misuse notes and a gaming check

Key concepts

The four key metrics: The delivery measures of the DevOps Research and Assessment (DORA) programme, divided into throughput and instability. They describe the change pipeline, so they belong chiefly to O3 Change, and they count only what the pipeline sees.
Missing data reads as good: An indicator computed from a telemetry pipeline has an implicit denominator, the events that arrived. When the pipeline fails, attainment looks healthy, so the chapter adds a data-completeness indicator beside it.
Distribution of time to restore: The reporting form both organisations adopt: the median, a high percentile, the count of incidents beyond the recovery time objective and their identifiers as a named list, with no mean by design, so that a tail incident stays visible.
Self-verification share: The share of an agent's actions whose only verification came from the executing agent or from a system sharing its model family and failure domain. Consumption of the autonomy budget is valid only when the verifying probe is independent, so the share must be zero at Level 3.
False green: A measure that reports health when the thing measured is unhealthy, for example through missing data counted as good or a definition that excludes the failure. Every measure therefore carries a gaming check.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Interactive exercise 1 · Match

Every measure carries a gaming check

Match each operations measure with the gaming check that the chapter's table gives it.

Interactive exercise 2 · Scenario

A request for figures per engineer

Apply the chapter's rules for metrics about people.

Teaching plan

15 min	The teacher introduces the rule that a metric is a question asked repeatedly and justified by a decision, then the four key metrics and the three limits the book reads into them.
15 min	Worked case: the tail incident INC-NR-2028-0391, the quarterly report that did not mention it, and the form that AIDR-0921 put in its place.
15 min	Participants take three rows of the table of metrics and state, for each, how it misleads and what the gaming check is.
35 min	Exercise: participants write metric definitions for their own operations report.
10 min	Debrief on metrics about people: subject, minimum group size of five and agreement before measurement.

Discussion

Which of the measures in your last operations report would have stayed green through the worst incident of the past year, and why?
A randomised trial cited in the chapter shows that perceived and measured performance can differ. Where does your organisation report a perceived gain from AI as if it were measured?
Short approval times in review read as efficiency. How would you tell efficient review from review that has become approval?

Written exercise

Exercise · 35 minutes

Metric definitions with misuse notes

Take the three most used measures in your next operations report, or failed deployment recovery time, time to restore and autonomy-budget consumption at Hansa Mutual. For each, write a definition: one sentence naming the decider and the decision the measure informs; the clocks, with start and stop events matching any legal definition that applies; the source, which must be independent of the thing measured; the form of reporting, as a distribution or count where a mean would hide the tail; the subject, which is a service, pipeline, delegation or team and never a person; the known ways it misleads; and the gaming check. If no decider can be named, strike the metric.

Deliverable: Three metric definitions, each with decision, clocks, source, reporting form, subject, misuse notes and gaming check, plus a note on any metric struck.

Your work

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

By how much did the outage of 7 hours 5 minutes in INC-NR-2028-0391 move Nordvik Region's reported mean time to restore?
Show the answer
C. By 9 minutes, so the report did not mention the outage
The mean moved by 9 minutes, and the committee later saw that the report had not mentioned the outage. AIDR-0921 replaced the mean with the median, the 90th percentile and a named list beyond the RTO.
Why does the book say the four key metrics are not a measure of the recovery performance of operations?
Show the answer
A. They describe the change pipeline, and failed deployment recovery time is not the time to restore of all incidents
An outage from a storage controller, an expired certificate or a provider's DNS is outside failed deployment recovery time, and changes made outside the pipeline are not counted. The metrics come from the research programme, not from the Regulation.
How does Nordvik Region report its change fail rate for a quarter with 62 production deployments and three failures?
Show the answer
D. As counts, because the denominator is too small to bear a rate
Rates need denominators large enough to bear them. The gaming-check table says to report counts under 100 deployments a quarter, and the Region reports 62 deployments and three failures as counts.