Observe with AI: Correlation, Anomaly Detection and Alert Noise

All courses / AI Operations / Module 2 / Chapter 6

Learning objectives

After the lesson the participant can:

Describe four capabilities AI adds to the alert stream and the failure mode of each
Distinguish the two honest forms of noise reduction from suppression that hides signal
Design an evaluation with a golden incident set, per-class recall, a noise budget, seeded alerts and sampling
Draft an alert-correlation policy with a suppression ceiling and a stop condition written as numbers

Key concepts

Baseline drift to the new broken normal: A baseline that adapts to the data adapts to degradation as well: a slow memory leak or creeping latency moves the baseline with it, and the anomaly score stays low while the service gets worse.
Golden incident set: Past operational episodes, each with the raw alert stream as it arrived and a ground truth written by people. It is held outside the system under evaluation, changed only by people who do not own the correlator, and every replay is logged.
Noise budget: A statement of how much unneeded interruption a rota may bear, made of three numbers: a ceiling on pages per on-call shift, a floor on the share of pages marked actionable, and a ceiling on out-of-hours pages that were not Tier 1. It is reported for the rota, never a named engineer.
Seeded alerts: Synthetic alert events introduced by an authorised injector upstream of the correlation layer, tagged with a run identifier and a sealed expectation of what should reach whom and by when, written before the injection and held outside the correlator.
Suppression ceiling: The rule that no alert on a customer-facing service level objective, on the heartbeat that watches the observation path or on a Tier 1 recovery objective is suppressed above Level 1 Assisted: the layer may propose and a person acts.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Interactive exercise 1 · Sort

What a correlation policy permits

Sort each behaviour of an alert-correlation layer by whether the three recurring rules of the chapter's correlation policy permit it.

Interactive exercise 2 · Match

Five instruments of evaluation

Match each instrument for evaluating an alert-correlation system with what it is, as the chapter describes it.

Teaching plan

15 min	The teacher presents the four capabilities and the cost of noise, then the two honest forms of noise reduction, deletion at the source and grouping with members kept.
15 min	Groups each take two of the six failure modes and explain why the chapter calls them failures of silence, using the Hansa Mutual ingestion failure as the case.
15 min	Reading of the evaluation record EVL-HM-2028-03: participants say what was tested, when, with what result, and what the missed seeded alert shows.
35 min	Exercise: participants draw up an alert-correlation policy for one service.
10 min	Debrief on what the published evidence does and does not establish, and on Nordvik Region's position that provider summaries are a reading aid.

Discussion

A model that learns which alerts engineers close without action will learn to withhold them. How would you find out that it had also withheld the rare alert that mattered?
The chapter treats a figure from another organisation's estate as a reason to try a method, not a measurement of it. What would your first own measurement be?
Hansa Mutual declares the vendor's correlation logic and measures it from outside until the contract grants test rights. What contractual terms would turn a declared feature into a verified one?

Written exercise

Exercise · 35 minutes

An alert-correlation policy for one service

For one Tier 1 or Tier 2 service, list its alert classes by consequence, not by tool: customer-facing objective burn, dependency health, saturation, batch and schedule, certificate and licence expiry, and hygiene. For each class state what the correlation layer may group, summarise, hold or suppress and at which degree of delegation, keeping suppression of customer-facing objective alerts at Level 1 Assisted at most. Make every hold expire in minutes. Add a noise budget for the rota, a weekly sampling rate for held-back alerts, a seeding plan with sealed expectations, and stop conditions written as numbers that a component outside the correlator can enforce.

Deliverable: A policy file with classes, actions and levels, a suppression ceiling, hold expiries, a noise budget, sampling and seeding rates and numeric stop conditions.

Your work

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

Which of these is the chapter's example of a stop condition that can actually be enforced?
Show the answer
B. Seed catch rate below 0.90 over four weeks freezes promotion; below 0.80 returns every class to Level 1
The third rule of a correlation policy is that stop conditions are numbers. A phrase such as reduce if quality falls is not a stop condition; thresholds on the seed catch rate can be enforced by a policy decision point the correlator cannot reach.
Where must the golden incident set be held, according to the chapter?
Show the answer
D. Outside the system under evaluation, changed only by people who do not own the correlator
These are held-out conditions: a system tuned on its own test set passes it. The set is kept out of the tuning data and the retrieval index, and every replay is logged.
What did the 28-day trial recorded in EVL-HM-2028-03 report for the customer-facing objective class?
Show the answer
A. All 14 episodes paged within the time budget, with none suppressed by AI
The record shows recall of 1.00 on 14 customer-facing episodes and zero suppressed by AI. The miss, SEED-HM-07, was in the dependency-health class, and median pages per shift fell from 31 to 19.