Case Studies and Anti-Patterns

All courses / AI Operations / Module 7 / Chapter 28

Learning objectives

After the lesson the participant can:

• Apply the four lenses of function, action class, observer principle and evidence to a post-incident report
• Explain the common pattern in the change cases, the recovery cases and the provider outages
• Name the eight anti-patterns, the function each breaks and the control that answers it
• Draft an anti-pattern record for one's own organisation with a control and a re-test date

Key concepts

Four lenses

The red threads of the book used to read a case: which function was concerned, which action class and what autonomy-budget reading applies, which observer test failed, and what the evidence behind any claim was. A reported cause is read as a statement of contributing conditions.

The green dashboard

Anti-pattern 1, which breaks O1 Observe: silence or missing data reads as health. The control is that no data counts as red and that an outcome probe runs in a separate failure domain.

The self-healing mask

Anti-pattern 2, which breaks O4 Restore: automatic repair hides the fault or prolongs it. The controls are independent verification, a budget counted by the observer and alert changes held at Level 0.

The first ring skipped

Anti-pattern 7, which breaks O3 Change: a change reaches its whole population before anything independent has judged it. The control is that every runtime change passes the first ring and flags are off by default.

The claim at the wrong scale

Anti-pattern 8: a recoverability claim that was configured and never restored, or true for one customer and untested for hundreds, fails at a dimension it never named, such as a client version, a scale or a location.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. The book's reading is that an AI agent in place of the script or pipeline in these cases would not have changed the mechanism. What would it have changed, and what does that imply for where controls belong?
2. Which of the eight anti-patterns would an honest review find in your own estate today, and what evidence would the review need to see?
3. Nordvik Region cannot run eight checks every quarter with seven engineers, so it re-tests its three records. How should a small team choose which patterns to test first?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. What pattern does the chapter find in the three recovery cases of GitLab, Atlassian and OVHcloud?

   1. A small trigger travelled by a path built for speed and reached every region
   2. The exposure was the whole population before anything independent had judged the effect
   3. A claim of recoverability was asserted and failed at a dimension it never named
   4. An agent acted beyond its mandate and gave a wrong account of its actions
   Show the answer

   C. A claim of recoverability was asserted and failed at a dimension it never named
   The recovery cases failed at a client version, a scale and a location: dimensions the claims never named. The first two options describe the provider outages and the change cases.

2. Which control does the table of anti-patterns give for the dead-man's switch that never fires?

   1. A heartbeat to a receiver outside the platform, with a silence rule
   2. Per-run tokens and a policy point outside the agent
   3. A named owner, a review date and a test of the written runbook
   4. Every runtime change passes the first ring, with flags off by default
   Show the answer

   A. A heartbeat to a receiver outside the platform, with a silence rule
   Anti-pattern 6 breaks O1 Observe when the alarm path shares the platform or nobody reads its messages. The control is a heartbeat received outside the platform and a rule that silence raises an alarm.

3. In the table of what the cases show, which observer test failed in the CrowdStrike case?

   1. Dependence: the failure email was rejected and nobody was told
   2. Stop: the tools needed to halt the change were on the failed network
   3. Dependence: customers' monitoring ran on the same cloud
   4. Independence: the validator and the code shared an assumption
   Show the answer

   D. Independence: the validator and the code shared an assumption
   The validator and the sensor code both assumed the same number of inputs, so the check shared its failure mode with what it checked. The other options describe GitLab, Meta and Google Cloud.