Restore: Runbooks, Auto-Remediation and Recovery

All courses / AI Operations / Module 2 / Chapter 10

Learning objectives

After the lesson the participant can:

• Write a runbook manifest with action class, level, preconditions, independent probe, rollback or hold, stop and owner
• Classify remediation actions by tier and assign a defensible ceiling to each
• Explain why a durable workflow engine does not guarantee that an effect happens once
• Argue both the case for concern and the case for calm on autonomous remediation, and state the chapter's verdict

Key concepts

Runbook manifest

The runbook written so that a policy engine, a person and an agent read the same thing: an RBK record with a trigger, an action class, a level, explicit preconditions, an independent probe, a rollback or hold, a stop and an owner with a review date.

Four remediation tiers

Distinguished by who acts and who approves: the system notifies (Level 0 Advisory), proposes while a person executes (Level 1 Assisted), executes each action with a named person's approval (Level 2 Supervised), or executes a catalogued runbook within a mandate (Level 3 Bounded).

Self-verification

The common failure in which an agent restarts a process and checks the process's own health endpoint: it has verified that the process answers, not that the service is better. Verification must be independent in identity, data path and failure domain, with a soak period.

Retry storms and duplicate effects

Retries make engines resilient and are also the mechanism of failure. Engines guarantee that a procedure progresses, not that its effect happens once, so idempotency keys, retry limits with back-off and a counter held outside the engine are conditions of delegation.

The last line

Backup and restore when remediation and rollback fail: offline or immutable copies, a segregated restore target, a restore path that does not depend on the failed system, and rehearsed restores. The cut-over of a restore over existing data is a named person's decision.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. A restart that clears a condition also clears the evidence. What must a runbook preserve before it acts, and who needs that evidence afterwards?
2. The case for calm argues that a bounded agent is more accountable than the alternative. What does an action under a manifest leave behind that a person at a console at night does not?
3. Why does the chapter treat restore over existing data as a named person's decision even when the agent can prepare the target and run the integrity checks?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. How does the chapter treat a request that an agent silence, mute, re-threshold or disable an alert, a probe or a check as part of remediation?

   1. Delegated at Level 3 Bounded when a catalogued runbook includes the step
   2. Delegated at Level 2 Supervised with approval by the on-call engineer
   3. Delegated at Level 1 Assisted during an authorised change window
   4. Not delegated at any level
   Show the answer

   D. Not delegated at any level
   Such an action removes the means by which the other safeguards are checked. It heads the list of actions an agent may not take and is not delegated at any level.

2. What is the chapter's verdict on unattended recovery of whole estates, in which agents diagnose, choose, execute and verify across services?

   1. Observable, since restarts and scale-out have been automated for years
   2. Expected, wherever a run record supports widening to Level 3
   3. Speculation, on the evidence examined
   4. Prohibited, by the four Level 4 prohibitions of Chapter 4
   Show the answer

   C. Speculation, on the evidence examined
   Bounded remediation of narrow reversible classes is observable, widening to reconfigure, roll back and perhaps failover is expected where evidence is recorded, and unattended recovery of whole estates is speculation.

3. In the split between engine and policy decision point, where must the stop for a runbook run ultimately be enforced?

   1. In a guard step of the workflow, editable by whoever edits the workflow
   2. By revocation of the credential and a deny entry, outside the platform
   3. By the engine's own cancel signal alone, since it is part of the engine
   4. In the agent's prompt, as an instruction to cease further actions
   Show the answer

   B. By revocation of the credential and a deny entry, outside the platform
   The engine has a cancel signal, but the rule is that an engine may enforce only a control that its operators alone can edit. The stop is backed outside the platform by revoking the credential and a deny entry.