Change, Restore and Runbook Tools

All courses / Tools / Module 5 / Chapter 16

Learning objectives

After the lesson the participant can:

• Define a standard change class with eligibility, verification, rollback and a suspension rule.
• Design a progressive-delivery policy with rings, bake times, stop conditions and an independent verifier.
• Write a runbook manifest that a policy decision point can enforce and a person can audit.
• Assign a ceiling per degree of delegation to change and restore actions and state what is required outside the agent.

Key concepts

Change class and change record

A class states what may be released without a person in the path; the record, family CHG, carries author, reconciler, computed class, plan summary, independent approvers, tested rollback and verification. The class is computed from the diff and the target.

Progressive-delivery policy

An artefact, family ROL, defining rings, bake times, stop conditions and the stop path for one service, and naming a verifier independent of the author, the pipeline and the telemetry path. The first ring is never skipped.

Runbook manifest

An artefact, family RBK, that makes a runbook executable under a bound: preconditions, dry run, action, independent verification, failure path, who may stop it and when it was last exercised. It is signed on release.

Remediation tiers and autonomy budget

Four tiers: notify at Level 0 Advisory, propose at Level 1 Assisted, execute with approval at Level 2 Supervised and execute within a mandate at Level 3 Bounded. Each Level 3 class carries a tolerance for faulty actions per 28-day window.

Sync policy as a mandate

The reading in AI Operations of a GitOps engine's settings (automated sync, self-heal, prune, sync windows, suspend) as ceilings on what the reconciler may do alone. A reconciler that applies is an actor in the change system.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. The Argo CD manifest of Architecture as Code enables prune on a production namespace, which the later ceilings of AI Operations would not permit. How should a team treat an older template when a later volume tightens the rule?
2. A progressive-delivery controller can abort a rollout but should not be the only judge of it. Who or what is the independent verifier in your own delivery path?
3. The runbook example allows restarting one of five instances but not the last healthy one. What does it mean that preconditions protect capacity and not only correctness?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. When, according to the practical guide of AI Operations, is a kind of change eligible to become a standard change class?

   1. When an agent has drafted it once and a named reviewer, independent of the requester, has approved the draft in a pull request
   2. When the change advisory board has not met to discuss that kind of change for twelve months
   3. When the supplier's documentation describes the change as safe to automate without any approval
   4. When it was made by hand at least 20 times in twelve months with no change-correlated Tier 1 or Tier 2 incident
   Show the answer

   D. When it was made by hand at least 20 times in twelve months with no change-correlated Tier 1 or Tier 2 incident
   The guide's condition for use is at least 20 manual executions in twelve months with no change-correlated Tier 1 or Tier 2 incident. The class then turns that history into data a gate can evaluate.

2. How does the change gate treat a change whose declared class is lower than the class computed from its diff?

   1. It denies the change, because the declared class is a claim and the computed class is the fact
   2. It accepts the declared class, because the author knows the change best
   3. It releases the change to the first ring only and skips verification
   4. It asks the agent that authored the change to reclassify it
   Show the answer

   A. It denies the change, because the declared class is a claim and the computed class is the fact
   A change is classified by the diff and the target, not by what its author says. The illustrative gate denies a change whose declared class is lower than its computed class.

3. Which ceiling does Table 16.1 set for prune in a GitOps sync policy?

   1. Level 3 Bounded for any class that passed the gate
   2. Any named holder, with no approval needed
   3. A named person for stateful kinds, and never Level 4 where it would delete data
   4. Level 4 Autonomous outside a freeze window
   Show the answer

   C. A named person for stateful kinds, and never Level 4 where it would delete data
   Prune deletes what left Git. The ceiling in AI Operations is a named person for stateful kinds, and never Level 4 where it would delete data. The no-approval holder applies to suspend.