The Observer Principle: Who Watches the Watcher

All courses / AI Operations / Module 4 / Chapter 17

Learning objectives

After the lesson the participant can:

State the three tests of the observer principle as questions answerable from records
Identify which kind of independence is missing in a described verification arrangement
Design an observer with a separate failure domain, an outcome probe and a heartbeat with a silence rule
Draft an observer manifest and a plan for an authorised, scoped seeded-fault test with a catch rate

Key concepts

Six kinds of independence: Model, data, authorship, credential, failure domain and time. The kinds are not ranked: a second model family used by the same author on the same metric gives model independence and nothing else, and a report says which kinds are present.
Dead-man's switch: A heartbeat that raises an alarm on silence. An observer that reports only when something is wrong cannot be told from an observer that has failed, so the absence of its signal must itself page someone.
Synced is not served: A reconciler's sync and health status is the engine's own view of its own work. Whether the application works for its users is shown by an outcome probe, and the engine's lag and suspension are watched from outside.
Seeded fault: A known weakness introduced on purpose by a person, under written rules of engagement, to test whether the observers see it. Each class has a catch rate by quarter, the share found within the target time, and the catch rate carries a stop rule.
Reproduction by a person: The final form of independence: a person outside the chain can redo a result from the evidence, without the agent, and reach the same conclusion. By default it is done for every claim that raises a level, every closure of a high-severity seeded-fault finding and a 10 per cent sample of other verifications.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Interactive exercise 1 · Sort

Which test of the observer principle fails?

Sort each arrangement by the test of the observer principle that it fails.

Interactive exercise 2 · Match

What the watcher shared with the failure

Match each public case with what, in the chapter's table, the verifier, observer or stop shared with the failed system.

Teaching plan

15 min	The teacher states the three tests and shows a loop that agrees with itself: an agent restarts, the same agent confirms, and a dashboard on the same platform reports green.
25 min	Five groups each take one row of the case table and report what was meant to verify, observe or stop, what it shared with the failed system, and which test failed.
20 min	Presentation of the six kinds of independence and the four devices of an observer outside the observed; participants classify the arrangement in the seeded-fault record SDF-HM-2028-002.
20 min	Debate in four steps on whether independence costs more than the loop saves, ending with the verdict by function and action class.
30 min	Exercise: participants specify an observer for one service.
10 min	Debrief on the table across the degrees of delegation: what verification, observer, stop path and human reproduction each level requires.

Discussion

None of the five public cases involved an AI agent. Why does the chapter say they are the pattern that an agent inherits and speeds up?
A verifier may share a model family with the agent it checks without harm for some claims. For which claims does shared model family matter, and for which is a script the better verifier?
If a model can recognise that it is in a test, behaviour under test may differ from behaviour in use. What follows for how much weight you place on an agent's own report and on evaluation alone?

Written exercise

Exercise · 30 minutes

An observer manifest for one service

For one Tier 1 service on which an agent or an automation acts, yours or Nordvik Region's portal under OBS-NR-0001, list the action classes with their levels, starting with the highest level and the least reversible class. For each class name the signal that shows the action worked, who or what produces it and by which path, and mark which of the six kinds of independence it has. Place the observer in a failure domain separate from the platform, define an external synthetic transaction and a heartbeat with a silence rule, and describe the stop path and its drill. Plan one seeded fault under written authorisation, in a scoped environment, with a target detection time and a named person who can stop the test.

Deliverable: An observer manifest with verifying signal and independence kinds per action class, a domain map, heartbeat rule, drilled stop path and one authorised seeded-fault plan.

Your work

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

In the chapter's case table, what did the CrowdStrike Content Validator share with the code it was meant to check?
Show the answer
C. The assumption of 21 inputs, held by validator and tested code alike
The validator evaluated the template on the same expectation as the code it tested, so the check shared its failure mode with the thing checked: a failure of test 1, independence of verification.
What did Hansa Mutual's game day of 2028-10-24 find about its remediation agent and the verification step?
Show the answer
A. They shared a model family, an identity and a signal
The agent's own check said the restart had worked while a probe the agent could not alter said otherwise. AIDR-0960 then kept restart at Level 3 Bounded only on condition of the observer manifest OBS-HM-0001.
According to the table across the degrees of delegation, what does the stop path require at Level 3 Bounded?
Show the answer
D. No approval, outside the control plane, drilled each quarter within the target
At Level 3 the stop needs no approval, runs outside the control plane and is drilled each quarter within its target. A stop that depends on the system under stress is a stop in name.