AI Operations · Module 2 · The Craft

Observe: Telemetry, Observability and Service Level Objectives

The lesson treats O1 Observe without AI, as the baseline that later chapters change. Participants learn why the unit of observation is the user journey, how a signal becomes a service level indicator, an objective and an error budget, how multiwindow burn-rate alerting works and where it does not fit, and what cost, retention and the GDPR mean for telemetry. They write an objective record for one service.

Lesson 5 of 31Chapter 590 minutes

Read the chapterThe whitepaperThe slides

All courses / AI Operations / Module 2 / Chapter 5

Learning objectives

After the lesson the participant can:

Key concepts

Symptom and cause
A symptom is what a user or calling system experiences, such as failing requests; a cause is why. The chapter follows the SRE literature in paging on symptoms tied to an objective and using cause-level signals for diagnosis.
Error budget
The complement of the objective over the window: the amount of failure the objective tolerates. At 99.9 per cent over 30 days a service may be unavailable for 43.2 minutes, at 99.95 per cent for 21.6 minutes. It is not the autonomy budget of Chapter 4.
Burn rate
How fast a service consumes its error budget relative to the objective. Alerts combine a long window, which confirms that significant budget is gone, with a short window, which confirms the burn is still going on.
Synthetic probe outside the platform
A scripted request that exercises a user journey at a fixed interval from a place the operator chooses. It gives steady traffic on a quiet service, measures what a user sees and keeps working when the platform's own telemetry fails.
Minimisation at the collector
The chapter's reading of GDPR Article 5(1)(c) and (e) for telemetry: attributes that identify people are deleted in the collection pipeline before storage, because not collecting potentially sensitive data is the best prevention.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Interactive exercise 1 · Put in order

Writing an objective from a user journey

Put the steps of the chapter's practical guide for writing a service level objective in order.

Interactive exercise 2 · Scenario

A quiet service that pages on noise

Apply what the chapter says about objectives where burn-rate alerting does not fit.

Teaching plan

15 minThe teacher contrasts monitoring that asks whether machines are busy with observing whether the service is good for its users, using the stalled claim ranking of INC-NR-2026-0214 that an availability probe would have missed.
20 minWorked calculation on the board: the error budget of SLO-NR-002 and SLO-HM-004, then the three rows of the burn-rate table and why 2 per cent of budget in one hour is a burn rate of 14.4.
10 minDiscussion of where burn rate does not fit, using Hansa Mutual's settlement run deadline, and of cost, tiering and personal data in logs.
35 minExercise: participants write an objective record from a user journey.
10 minPeer review against the definition of done, with attention to what each indicator cannot see.

Discussion

  1. The chapter says an operations function that cannot say when it is blind has not observed anything. How would your organisation learn that its observability platform had stopped ingesting?
  2. Why does the chapter warn against a service that is far more reliable than its objective, and what follows for what you promise to dependent services?
  3. A metric by named engineer is a measure of a person. Where does your telemetry record people at work, and who has agreed to it?

Written exercise

Exercise · 35 minutes

An objective record from a user journey

Choose one service, your own or Nordvik Region's citizen e-service portal. Name in one sentence, from the user's side, the one or two journeys whose failure is worth paging for. Define an indicator for each as a share of good events, set an objective and a window, and calculate the error budget in minutes. Add a page-level burn-rate alert with a long and a short window, a measurement point outside the platform, the owner role and the on-call route. State how the page will be tested with a seeded fault, and write one sentence on what the indicator cannot see.

Deliverable: A versioned objective record with indicators, objectives, error budget, a two-window burn-rate alert, an outside probe, an owner, a planned alert test and a stated blind spot.

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

  1. In the chapter's starting parameters for multiwindow alerts, which burn rate pages a person on a one-hour long window with a five-minute short window?

    Show the answer

    C. A burn rate of 14.4, consuming 2 per cent of the budget
    In a 30-day window of 720 hours, 2 per cent of the budget consumed in one hour is 0.02 times 720, a burn rate of 14.4. The rate of 6 belongs to the six-hour window and 1 to the three-day ticket.

  2. Why does a ratio-based error budget degenerate for Hansa Mutual's objective that the settlement run completes by 06:00 on run days?

    Show the answer

    A. With about twenty run days a month, 99.95 per cent allows about one hundredth of a missed run, which is none
    A deadline objective is binary per run day. With perhaps twenty run days in a month the budget allows no missed run at all, so the ratio method gives nothing to burn and another form of alert is needed.

  3. What does the recovery block of the record SLO-NR-002 deliberately show beside the promised recovery time of 2 hours?

    Show the answer

    D. The failed test RST-NR-2027-002, which measured 5 hours 20 minutes and has no re-test
    The record states the promised objective and shows the evidence beside it. Because the test of 2027-10-12 failed and no re-test exists, the claim that the portal is recoverable within two hours is not made.