All courses / Tools / Module 5 / Chapter 15
Learning objectives
After the lesson the participant can:
- Write a service level objective for a user journey with an error budget and a budget policy.
- Calculate the failure ratio at which a multi-window burn-rate rule fires for a given objective.
- Draft an alert-correlation policy with a never-suppress list, sampling and stop conditions.
- Define a metric with the decision it informs, its population, its clock and its known misuse.
Key concepts
- Service level objective record
- A record, family SLO, of what users are promised and how it is measured, alerted on and evidenced, including burn-rate alerts, the budget policy and the evidence that the alerts work. It is reviewed when the service, gateway or collector changes.
- Burn-rate alerting
- Multi-window alerting on the speed at which the error budget is consumed. A burn rate of 1 exhausts the budget exactly over the window; a rule fires at the burn rate multiplied by the permitted failure ratio.
- Watchdog alert and observer heartbeat
- An alert that is always firing, so that its absence at a receiver outside the platform shows that the alerting pipeline is broken. AI Operations generalises it: every observer emits a heartbeat to an external receiver.
- Alert-correlation policy
- An artefact, family ACP, that sets per alert class what a correlation layer may group, suppress or page, what it may never suppress, the sampling by a person and the seeded-miss rates that freeze its promotion. Its mandate is a ceiling.
- Metric definition catalogue
- Entries, family MET, that define each metric with the decision it informs, its population, clock, source and subject, a minimum group size, misuse notes and gaming checks. A report function shows a metric without a mean.
Interactive exercises
Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.
Interactive exercise 1 · Put in order
Seven links from signal to person
Put the tools of Chapter 15 in the order of the chain from signal to person that the chapter's Summary gives.
Interactive exercise 2 · Scenario
Switching on a noise-reduction feature
Choose what the head of site reliability should do before the feature acts, following the practical guide on the alert-correlation policy.
Teaching plan
| 15 min | Present Figure 15.1, from telemetry to page. Cover OpenTelemetry, the collector that separates a sampled operational pipeline from an unsampled evidence pipeline, and the service record as anchor. |
| 20 min | Work through the objective record template and Table 15.1 of burn-rate parameters. Participants compute the one-hour page threshold for an objective of 99.9 per cent. |
| 15 min | Teach correlation and measurement: the policy ACP-HM-2028-01, the evaluation against a golden set held outside the correlator, the noise budget and a metric catalogue entry. |
| 30 min | Exercise: participants write an objective record and a correlation policy outline for one service. |
| 10 min | Debrief: where is each objective measured from outside, and what evidence shows that its alerts work. |
Discussion
- The chapter says a metric that an agent reports about itself is a claim and not a measurement. Which figures on your operations dashboard are of that kind?
- The noise budget judges an alerting estate by what it costs the rota, not by what it detects. What would change in your alert reviews if that were the measure?
- The abbreviation DORA names both a regulation and a research programme, and the volumes differ on the number of key metrics. How should a report to a board handle such ambiguity?
Written exercise
Exercise · 30 minutes
An objective record and a correlation policy for one service
Choose one service with a critical user journey in your organisation, or the citizen e-service portal of Nordvik Region. Write a service level objective record in outline: the indicator for the journey, the objective and window, the measurement point inside and the probe outside the service, the page and ticket burn-rate rules with long and short windows from Table 15.1, the budget policy, the owner and the review date, and the evidence that the alerts fire. Then outline an alert-correlation policy for the same service: three alert classes by consequence, what may be grouped or suppressed and at what level, the never-suppress set, the weekly human sample and two stop conditions.
Deliverable: An objective record outline with burn-rate rules and alert evidence, and a one-page correlation policy with never-suppress set, sampling and stop conditions.
Self-test
Choose an answer to see whether it is right, or open the answer below each question.
In the starting parameters of Table 15.1, which burn rate pages on a one-hour long window with a five-minute short window?
Show the answer
C. 14.4, consuming 2 per cent of the budget
The table gives a page at burn rate 14.4 over one hour with a five-minute short window, consuming 2 per cent of the budget; 6 belongs to the six-hour page and 1 to the three-day ticket.What ceiling does the worked alert-correlation policy set for suppressing a customer-facing objective alert?
Show the answer
B. It is never suppressed above Level 1 Assisted
The mandate in the record is a ceiling, not a recommendation: a customer-facing objective alert is never suppressed above Level 1 Assisted, a hold expires, and stop conditions freeze promotion.Why does the telemetry collector configuration of AI Platform and Security keep an unsampled evidence pipeline beside the sampled operational one?
Show the answer
A. The stores have different purposes: the operational one is cheap to query and ages out, the evidence one is a full record
The configuration separates two stores with different purposes: a sampled operational pipeline that is cheap to query and ages out, and an unsampled record of events routed to an append-only store, both after redaction of personal data.