What Operations Did Before: Runbooks, ITIL, DevOps, SRE and AIOps

All courses / AI Operations / Module 1 / Chapter 2

Learning objectives

After the lesson the participant can:

Name the six operating models of the lineage and the scarcity each answered
Explain the survey finding on external change approval and the three cautions the chapter attaches to it
Compare a ticket-era and a code-era change record by what each can prove
Classify the practices of one service by lineage, scarcity and disposition

Key concepts

Scarcity behind the shape: The book's own interpretation that each operations practice is a device for rationing attention: the ticket queue rations order, the runbook rations expert judgement, the rota rations hours, the change advisory board rations review, and the alert threshold rations the operator's gaze.
From scripts to state engines: A second lineage of technique that moves from describing steps to describing a desired state and from running on command to continuous reconciliation, as in Kubernetes controllers and GitOps; each step moved the loop from the person to the engine.
Toil: Work tied to running a service that is manual, repetitive, automatable, tactical and without enduring value, and that scales linearly as the service grows; the SRE literature reports an aim of keeping it below 50 per cent of each engineer's time.
First-generation AIOps: The application of machine learning to operational data, answering scarcity of attention to signals. Research concentrated on O1 Observe and O2 Diagnose, and remediation was the least studied area; practitioners listed label scarcity, data quality and misbehaving models among its difficulties.
Ironies of automation: Bainbridge's 1983 argument from industrial control: automating a task leaves the operator the tasks that could not be automated, which are monitoring the automation and taking over when it fails, while the skills needed for taking over decay.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Interactive exercise 1 · Match

Operating models and their artefacts

Match each operating model in the lineage with its characteristic artefacts, as the chapter's side-by-side table gives them.

Interactive exercise 2 · Sort

Delegation that was already there

Sort each practice that predates AI by the nearest degree of delegation in the chapter's table. The mapping is the book's own reading.

Teaching plan

20 min	Presentation of the six models side by side: scarcity answered, characteristic artefact and what remained to a person, followed by the second lineage from scripts to state engines.
10 min	Close reading of the change advisory board finding: what the survey reported, what it recommended instead, and the three cautions on survey evidence.
10 min	Participants compare the two Hansa Mutual change records, CHG-HM-5870 and CHG-HM-6101, and list what each record can prove.
25 min	Exercise: participants build a practice register for one service.
10 min	Debrief using the table across the degrees of delegation: which existing practices are already close to Level 3 Bounded in shape, and what was usually missing.

Discussion

Which practice in your operations function has outlived the scarcity that produced it, and what would be lost if it were retired tomorrow?
The chapter says AI lowers the price of reading but not the price of deciding or of being accountable. Where does that distinction fall in your incident process?
A watchdog script that restarts a failed process is described as close to Level 3 Bounded in shape. What does it lack that the book would require of an agent doing the same thing?

Written exercise

Exercise · 25 minutes

A practice register for one service

Following the practical guide, list the practices in use for one service on one line each: the queue and its priority rules, each runbook, the on-call rota, the alert rules that page, the change types and their approval routes, and the incident and problem reviews. For each entry record the lineage it comes from, the scarcity it answered, when it was last used, an owner, whether the scarcity still holds and a disposition: keep, automate, change owner or retire. The scope is practices and not people, so no entry may name a person as a performance measure.

Deliverable: A register of at least eight practices with lineage, scarcity answered, last use, owner, disposition and a first review date.

Your work

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

What did the 2019 delivery research report find about requiring approval by an external body, such as a change advisory board, for significant changes?
Show the answer
A. Respondents with it were 2.6 times more likely to be low performers, with no evidence of lower change fail rates
The survey reported the 2.6 times association and no evidence of lower change fail rates, and recommended peer review supported by automation. The chapter adds that this is an association from survey responses, not an experiment.
In the ticket-era record CHG-HM-5870, what does the chapter say the approval was a property of?
Show the answer
C. The pre-approved standard change template
Because the change type was standard, nobody reviewed the content of this change and the test-evidence field was empty; the approval belonged to the template and not to the change, which the incident INC-HM-2027-0833 made costly.
Which scarcity does the chapter say site reliability engineering answered?
Show the answer
D. Reliability engineering skill
The side-by-side table gives reliability engineering skill for SRE, with the SLO, error budget, toil cap and blameless review as its characteristic artefacts. The other three answers belong to the sysadmin, DevOps and platform engineering.