Verifying What the AI Concludes: Review for Soundness When the Analyst Depends on the AI

All courses / AI Blue / Module 3 / Chapter 14

Learning objectives

After the lesson the participant can:

• Explain the three forms of dependence and why automation bias makes designed verification necessary
• Apply the six soundness criteria to a triage verdict, a timeline and a containment proposal
• Design a seeded-incident programme with a catch-rate threshold and a stop rule
• Interpret a set of oversight-quality metrics and propose measures that keep review real at volume

Key concepts

Soundness criteria

Six criteria defined before the conclusion arrives: grounding, completeness, alternatives considered, stated confidence, reversibility and stability. They belong in the mandate manifest and the closure policy so the pipeline rejects output that fails them.

Key fact

The one fact on which a conclusion depends. The reviewer reproduces it by a route the AI did not take, re-running the query by hand or confirming it from a second source the adversary could not have altered.

Critique agent

A second AI that reads the evidence and the first agent's conclusion and flags unsupported claims and missing alternatives. It runs on a different model family, raises questions and never gives an approval.

Caught-seed rate

Seeded faults caught over seeded faults injected, read with its counts. The programme starts with a threshold of 0.80 over a rolling quarter; below it, extension of delegation for the class is frozen.

Back-pressure

When a reviewer's cap of open proposals is reached, 12 by default, a Level 2 class drops to Level 1: the agent keeps summarising but stops proposing until the queue clears. Pressure goes to the agent, not the person.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. A model's stated confidence is part of its output and not a measurement of its accuracy. What should a reviewer accept as a basis for confidence instead?
2. With five seeds in a month a single miss moves the monthly rate by 0.20. How should small numbers be reported so that nobody reads more into them than they hold?
3. The chapter places accountability for a missed closure first with those who designed the review and then with the person who approved. What would your organisation need to record to make that distinction visible?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. What is the cheapest correction for anchoring, the strongest effect of automation bias, according to the chapter?

   1. Asking the agent to state its confidence as a number before the analyst reviews it
   2. Replacing the reviewer after every missed seed
   3. Doubling the number of reviewers on each shift
   4. Changing the order of work: the analyst forms a provisional view before opening the AI's conclusion
   Show the answer

   D. Changing the order of work: the analyst forms a provisional view before opening the AI's conclusion
   The reviewer who reads the conclusion first sees the evidence through it. The analyst therefore opens the raw alert, writes a provisional verdict and the fact that would change it, and only then opens the summary.

2. Why does the critique agent run on a different model family from the agent it critiques?

   1. Because agents on one model family may share blind spots and fail together
   2. Because a second model family is cheaper to run at night
   3. Because the works agreement requires two suppliers for every tool
   4. Because the critique agent needs write access to the case that the first model lacks
   Show the answer

   A. Because agents on one model family may share blind spots and fail together
   The design choice draws on the correlated-failure argument: a second opinion from the same model family may repeat the first agent's mistake. The critique agent stays read-only and never approves.

3. In Hansa Mutual's oversight report for the third quarter of 2027, how does a reader learn that the caught-seed rate of 0.86 is fragile?

   1. The report omits the number of seeds injected
   2. The report notes beside the figure that two more misses would breach it
   3. The report ranks the analysts who missed the four seeds
   4. The report shows that the rejection rate fell to zero while volume tripled
   Show the answer

   B. The report notes beside the figure that two more misses would breach it
   With 28 seeds, 24 caught gives 0.86, but 22 of 28 is 0.79. The report states this beside the figure and also says that the reproduction rate and unassisted share missed their targets.