Cover of AI Operations

Book 10 · Course

AI Operations

Design, bound and verify an operations loop in which AI observes, diagnoses, changes and restores services, and show a board or supervisor the evidence.

7 modules31 lessons8 lecturesabout 50 hours of teaching

Start with Chapter 1The lecturesRead the book online

All courses / AI Operations

About the course

The course follows AI Operations, Book 10 of The Transformed Architecture. It begins with operations before AI and the pressures on it, then defines the operations loop with its five functions, O1 Observe, O2 Diagnose, O3 Change, O4 Restore and O5 Sustain, the autonomy budget and the observer principle. It covers the craft of each function, the operation of AI systems themselves, the operator's own agents as Tier 0, and the organisation around the loop: resilience as a claim, measurement, operating models, decision rights, European regulation and provider boundaries. It closes with the people who carry the pager, public cases, a year at two fictional organisations and the verdict. It matters because operations holds the broadest credentials in the estate and closes loops without a person in them.

After the course the participant can

  • Describe an operations practice or agent by function, action class, usage mode and degree of delegation
  • Set a ceiling per action class from reversibility, blast radius, evidence and window, and attach an autonomy budget to it
  • Apply the three tests of the observer principle to a verification, an observer and a stop path
  • Write the artefacts of the loop: an objective record, a correlation policy, a runbook manifest, a mandate manifest and an observer manifest
  • Write a recoverability claim with its six parts and derive its status from dated, attested tests
  • Assign the decision rights of an incident and map a regulatory duty to a control, an owner and evidence
  • Design an on-call charter and a drill year that keep the load humane and the manual craft in practice
  • Present to a board what the organisation can and cannot prove about its operations, with labelled statements about the future

Three ways to run it

The same material in three lengths. Each row of an agenda names the lessons it draws on.

Executive briefing90 minutes

For management bodies, chief information officers, heads of operations and risk committees who must decide how far operations agents may act

A briefing on the decisions that belong to the board. It states the five-part thesis, shows the frame of ceilings, budget and observer, and concentrates on what the organisation can prove about recovery and who may stop an agent. It ends with the five board decisions of Chapter 30 and the five things to do on Monday.

0:00Two nights, one outage: why the faster night is not the better one, and the five-part thesis Ch. 1
0:15The frame in ten minutes: five functions, a ceiling per action class, the autonomy budget and the four cases where Level 4 Autonomous is never used Ch. 4
0:30Who watches the watcher: the observer principle and the operator's AI as Tier 0 Ch. 17, Ch. 18
0:45Resilience as a claim: what the register lets the board say, and the duties of the management body under European regulation Ch. 19, Ch. 23
1:00Decision rights and the people who hold them: the stop right, the incident commander and the on-call charter Ch. 22, Ch. 25
1:15Five decisions for the board, the verdict on the thesis and questions Ch. 30, Ch. 31
One-day seminar1 day

For operations, site reliability and platform leads, incident commanders, architects and resilience officers who will write the artefacts

A working day that moves from the frame to the artefacts. Participants bring the records of one Tier 1 service and leave with a draft delegation profile, an observer design, a recoverability claim with its status and a decision-rights matrix. Lectures are short; most of the time goes to exercises from the lessons and to review in pairs.

09:00Opening lecture: from the queue to the control loop, the pressures and the frame Ch. 1, Ch. 3, Ch. 4
09:45Exercise: a delegation profile for one agent or automation on one service Ch. 4
10:45Observing and diagnosing with AI: objectives, suppression ceilings and the assistant as evidence Ch. 5, Ch. 6, Ch. 7
11:30Changing and restoring: provenance, the first ring, runbooks as code and the debate on autonomous remediation Ch. 8, Ch. 9, Ch. 10
13:15The operator's own AI: mandates, the observer principle and Tier 0 defence, with an exercise on an observer manifest Ch. 16, Ch. 17, Ch. 18
14:30Exercise: a recoverability claim, its derived status and the test behind it Ch. 19
15:15Decision rights, regulation and provider boundaries, with an exercise on the first ten decisions of a bad night Ch. 22, Ch. 23, Ch. 24
16:15People and craft: the on-call charter and the drill year Ch. 25, Ch. 26
16:45Closing: anti-patterns, the verdict and five things on Monday Ch. 28, Ch. 31
Full course7 weeks, one module a week

For teams that operate services with AI and the functions that govern, audit and negotiate them

The full course covers all 31 lessons in the order of the book, one part a week, with a module lecture, the lesson exercises and a capstone carried through the seven weeks on one Tier 1 service. Participants build the operations dossier step by step and present it in the last week as a board paper.

Week 1The operations in question: thesis, history, pressures and the loop; capstone service chosen and inventoried Ch. 1, Ch. 2, Ch. 3, Ch. 4
Week 2The craft: observe, diagnose, change, restore and sustain, with one artefact per function Ch. 5, Ch. 6, Ch. 7, Ch. 8, Ch. 9, Ch. 10, Ch. 11, Ch. 12
Week 3Operating AI systems: release manifests, AI incidents and their clocks, and the economics Ch. 13, Ch. 14, Ch. 15
Week 4The operator's own AI: agent team and mandates, the observer principle, Tier 0 defence Ch. 16, Ch. 17, Ch. 18
Week 5Operations in the organisation: claims, metrics, operating models, decision rights, regulation and boundaries Ch. 19, Ch. 20, Ch. 21, Ch. 22, Ch. 23, Ch. 24
Week 6People and career: on-call charter, competences and drills, and the horizon register Ch. 25, Ch. 26, Ch. 27
Week 7Experience and conclusions: cases and anti-patterns, the year, the board answers and the verdict; capstone presentations Ch. 28, Ch. 29, Ch. 30, Ch. 31

Modules and lessons

One module for each part of the book and one lesson for each chapter. A lesson has learning objectives, key concepts, a teaching plan, discussion questions, an exercise and a self-test.

Module 1 · Part I · The Operations in Question

Part I states the question of the book and builds its frame. The participant meets the five-part thesis, the lineage of operating models and the scarcity each answered, the six pressures that strain the inherited shape, and the operations loop with its five functions, eleven action classes, autonomy budget, Level 4 prohibitions, observer principle and resilience claim. The participant leaves able to describe any operations practice or agent in the book's vocabulary and to draft a first delegation profile. Lecture: From the Queue to the Control Loop: The Frame of AI Operations

Module 2 · Part II · The Craft

Part II is the craft: one or more chapters per operations function. The participant builds the artefacts that make the loop real, namely a service level objective with burn-rate alerts, an alert-correlation policy with a suppression ceiling, an incident record that marks every assistant contribution, a change gate and a standard change class, a rollout policy with an independent verifier, a runbook manifest, guardrails for a rightsizing agent and an action register with a closure rule. In each, what verifies is kept apart from what acts. Lecture: The Craft: Five Functions, One Rule About Who Verifies

Module 3 · Part III · Operating AI Systems

Part III turns the relation round and treats AI as the thing operated. The participant learns why an AI service changes when nobody has deployed anything, how a release manifest, quality objectives and a canary by case make that change visible and reversible, how incidents involving AI are classed, contained and reported under several clocks at once, and how cost, latency, quality and energy are budgeted with a stop that does not depend on the agent. Lecture: Operating AI Systems: Change Without a Deployment

Module 4 · Part IV · The Operator's Own AI

Part IV treats the operator's own AI: a team of narrow agents, each with a role, an identity, an owner and a mandate; the observer principle that says what must verify their work and what must watch the watchers; and the defence of the agents, their credentials, their runbook library and the control plane as Tier 0. The participant leaves with a mandate manifest, an observer manifest and a deny-by-default permission policy with ceiling cases. Lecture: The Operator's Own AI: Mandate, Observer and Tier 0

Module 5 · Part V · Operations in the Organisation

Part V places the loop in the organisation. The participant learns to write a recoverability claim and derive its status, to define metrics by the decision they inform and to spot the ones that mislead, to choose an operating model per service and buy operations with AI inside, to assign the decision rights of a bad night, to turn an article of European regulation into a control with an owner and evidence, and to test a provider boundary and an exit. Lecture: Operations in the Organisation: Claims, Measures, Rights and Boundaries

Module 6 · Part VI · People and Career

Part VI turns to the people at the end of every function. The participant learns what studies report about on-call load, alert fatigue and the supervision of automation, and how an on-call charter agreed with employee representatives answers them; which competences an operator needs and how the craft is kept with manual runs, dark days and seeded-error drills; and how to hold a labelled, falsifiable view of the future in a horizon register. Lecture: People and Career: Load, Craft and the Question of 2036

Module 7 · Part VII · Experience and Conclusions

Part VII tests the book against the record and draws its conclusions. The participant reads eleven public cases by mechanism and learns eight anti-patterns with their controls, follows Nordvik Region and Hansa Mutual through twelve months of applying the rules, studies the answers and twelve-month plans that go to each board, and weighs the verdict on each part of the thesis. The participant leaves with a board paper outline and five written records for a first week. Lecture: Experience and Conclusions: What the Record Shows and What Remains Open

Final assignment

Capstone

An Operations Dossier for One Tier 1 Service

Over the course, each participant or team builds the operations dossier of one Tier 1 service, either from their own organisation or one of the services of Nordvik Region or Hansa Mutual as the book describes them. The dossier states who and what may act on the service, at which degree of delegation, what verifies each action, who can stop it and what the organisation can prove about recovery. Every claim in it carries what was tested, when and with what result, and every unknown is written as unknown. Any test proposed in it is authorised, scoped and has a stop path outside the system under test. The dossier is presented in the final week as a paper to a board or risk committee, with statements about the future labelled observable, expected or speculation.

Steps

  1. Inventory where AI already takes part in operating the service, including provider features, and draw its dependency-and-blast-radius map with unknowns
  2. Write the delegation profile: a ceiling per action class from reversibility, blast radius, evidence and window, with the autonomy budget and the stop holders
  3. Specify the observer: the verifying signal per action class, the kinds of independence it has, its failure domain, heartbeat and the stop drill
  4. Write the recoverability claims of the service with their six parts, derive each status, and plan the test that the oldest or failed claim needs
  5. Build the decision-rights matrix for the first ten decisions of a bad night and the clauses of the on-call charter that concern the rota
  6. Present the dossier as a board paper with five decisions, gated milestones for twelve months, indicators with misuse notes and risks with triggers

Deliverables

  • A dossier of the service: inventory, dependency map, delegation profile, observer manifest, resilience register rows and decision-rights matrix
  • A plan for one authorised, scoped test, a restore test, a seeded fault or a stop drill, with roles, abort rule and attestation
  • A board paper of at most four pages and a ten-minute presentation of it

Assessment criteria

  • Faithful use of the book's frame: the five functions, the action classes, the degrees of delegation and the Level 4 prohibitions are applied correctly
  • Independence is shown, not asserted: for each delegated action the verifier, the observer and the stop path are named with what they do not share with the actor
  • Evidence carries an age: every statement about recovery, coverage or competence gives the test, the date and the result, or is marked unknown or untested
  • The paper is decidable: a board could take the five decisions from it, and statements about the future are labelled observable, expected or speculation
  • People are respected: metrics concern services and teams, on-call clauses note what needs agreement with employee representatives, and no monitoring of named staff is proposed