Change Safely: Progressive Delivery, Freezes and Blast Radius

All courses / AI Operations / Module 2 / Chapter 9

Learning objectives

After the lesson the participant can:

• Explain how rings, bake time, feature flags and cells limit the exposure of a change
• Identify, in a public incident report, why a configuration change was not contained
• Specify stop conditions and the properties of an independent verifier for a staged rollout
• Draft a rollout policy with a blast-radius budget and a freeze rule for one Tier 1 service

Key concepts

Change-correlated incident rate

The share of incidents in a period for which a change to the affected service or its dependencies fell inside a declared look-back window. A report states the window, the denominators and the changes it cannot see; correlation in time is not cause.

Ring and bake time

A ring is a stage of a rollout defined by the population that receives the change; bake time is the period a change stays at a stage, observed, before it is judged and widened.

Independent verifier

The system that evaluates stop conditions: it is not the releaser, reads from its own path, cannot be edited by the releaser, holds a guardrail out of sight and has been shown to fire by a dated seeded-fault test.

Freeze

A suspension of some or all change to a service for a period when the cost of failure is unusually high or the capacity to respond is low. It has a scope, a period, a reason and an owner, and named holders who may authorise a recorded exception; never an agent.

Blast-radius budget

A statement of how far a mistake may spread before a person must decide, with four parts: scope per ring, the longest time a change may stay unverified at that scope, the number of changes in flight at once, and the record of consumption.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. In INC-HM-2027-0833 a configuration push was classed as standard and the canary stage was skipped. What would have had to be different in the pipeline for the classification not to matter?
2. Nordvik Region's second look comes from a provider that shares its DNS dependency. How independent is that verifier, and what would a probe path that does not share the dependency look like?
3. A freeze that no one can break will be broken informally. Who should hold the exception in your organisation, and what must the record of an exception contain?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. According to the chapter's reading of the public reports, why is configuration the dangerous change?

   1. It is written mostly by agents, whose drafts contain more defects than code written by people
   2. It travels by the fastest path with the weakest checks and can reach a whole population at once
   3. It cannot be held in version control and therefore has no provenance record
   4. It is exempt from the change management duties of DORA and NIS 2
   Show the answer

   B. It travels by the fastest path with the weakest checks and can reach a whole population at once
   Code is usually protected by the longest pipeline and configuration by the shortest. In the CrowdStrike, Google and Cloudflare cases the change reached its entire population by a distribution path before anything independent had judged it.

2. In Hansa Mutual's design, how far may the IT-operations agent release a standard change on its own, and who decides to widen?

   1. To all cells; the agent widens when its own post-sync test passes
   2. To cell 2; the service owner widens after reading the agent's report
   3. To ring 0 only; the change advisory board widens at its weekly meeting
   4. To cell 1; the pipeline controller widens on the independent verifier's pass
   Show the answer

   D. To cell 1; the pipeline controller widens on the independent verifier's pass
   The agent can reach cell 1 and no further. Widening is a decision of the pipeline controller, conditioned on a verifier whose identity and age the agent does not write, so the widening decision stays outside the agent.

3. What do progressive-delivery controllers such as Argo Rollouts supply, and what do they leave open?

   1. They supply the mechanics of hold, abort and promote, and do not supply independence
   2. They supply the independent verifier, and leave the bake times and the order of rings to the releasing team
   3. They supply the freeze calendar, and leave rollback to the incident commander
   4. They supply the blast-radius budget, and leave stop conditions to the agent
   Show the answer

   A. They supply the mechanics of hold, abort and promote, and do not supply independence
   The controller's own analysis is a fast first gate. The verdict that permits widening comes from the independent verifier, and the analysis template and thresholds are stop conditions the releaser cannot edit.