Observability, Monitoring and Drift in Production

All courses / AI Platform and Security / Module 5 / Chapter 21

Learning objectives

After the lesson the participant can:

• Distinguish data drift, concept drift and performance decay and say which signals reveal each.
• Explain why the evidence pipeline is unsampled while the operational pipeline is sampled.
• Define a drift detection job with reference window, current window, checks and actions.
• Draft an AI incident runbook that narrows before it suspends and assesses notifications in parallel.

Key concepts

Data drift and concept drift

Data drift occurs when the distribution of inputs changes while the relationship between inputs and outputs stays the same. Concept drift occurs when the relationship itself changes. Performance decay is the observable consequence of either.

Evidence pipeline

The route by which the record of events reaches an append-only store with retention locks. It is not sampled, because Article 12 of the AI Act requires automatic recording of events over the system's lifetime.

Drift detection job

Scheduled, versioned code that compares a reference window with a rolling current window for input drift, prediction drift and delayed-label accuracy. Its thresholds are the same values as in the post-market monitoring plan.

Blind sample

A small sample in which officers decide before seeing the system's suggestion, with a record of whether each decision was made with or without it. Without it, accuracy measured against decisions that follow the suggestion becomes meaningless.

Impact query

A query listing which decisions each model version made in which period, used for remediation after an incident. It is the reason the Article 12 record must be complete and unsampled.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. Labels often arrive late or never. For which of your systems do you never learn the right answer, and what do you monitor instead?
2. Suspension is called an architectural requirement: a tested switch and a fallback that absorbs the load. Could your organisation process the work manually for a week, and has it tried?
3. Awareness time is recorded at detection because a time reconstructed later will be reconstructed generously. What else in an incident record is tempting to reconstruct afterwards?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. Why is the evidence pipeline in the collector configuration not sampled?

   1. Because Article 12 of the AI Act requires automatic recording of events, and a sample is not a record
   2. Because sampling would expose prompt and completion content that hashing in the collector cannot protect
   3. Because the observability back end cannot store traces that have been sampled
   4. Because unsampled traces are cheaper to query than sampled ones
   Show the answer

   A. Because Article 12 of the AI Act requires automatic recording of events, and a sample is not a record
   The event log store is evidence kept for the regulatory period; the operational trace store is sampled because its cost should match its purpose.

2. What does the chapter prescribe to counter a feedback loop in which case officers accept a decision support tool's suggestions by default?

   1. Retrain the model more often on the recorded decisions, so that it follows the officers' latest practice
   2. Hide the confidence score from the case officers
   3. Record whether the human decided with or without the suggestion, and keep a small blind sample
   4. Raise the alert threshold for prediction drift on the predicted route
   Show the answer

   C. Record whether the human decided with or without the suggestion, and keep a small blind sample
   Otherwise recorded decisions converge on the model's output, and the authority cannot show that human oversight is effective and not nominal.

3. In Nordvik Region's drift runbook, how may a drift threshold be changed?

   1. By the on-call engineer silencing the alert for the affected slice
   2. By the supplier, as part of the next quarterly retraining
   3. By the incident manager during containment, with a note in the ticket
   4. Through a pull request to the drift job, never a silenced alert
   Show the answer

   D. Through a pull request to the drift job, never a silenced alert
   Forbidding silenced alerts keeps the thresholds aligned with the post-market monitoring plan, which is generated from the same file.