Diagnose: Triage, Root-Cause Analysis and the Incident Assistant

All courses / AI Operations / Module 2 / Chapter 7

Learning objectives

After the lesson the participant can:

• Describe the four capabilities of the incident assistant and why its tools are read-only
• Explain what a published root-cause accuracy figure does and does not show for another estate
• Apply the second-hypothesis rule before a consequential action based on a ranked hypothesis
• Draft an incident record that separates the assistant's retrieval, ranking and drafting contributions

Key concepts

The assistant as evidence, not authority

The chapter's own conclusion: in diagnosis the assistant is a source of evidence and hypotheses and never the authority for an action. Its tool surface is read and annotate only, and a hypothesis is a reason to act, not an authority to act.

Second-hypothesis rule

Before an action of a class above annotate, notify or scale is taken on the assistant's top-ranked hypothesis, a person records a second hypothesis and one observation that would refute the first, from a source the assistant did not select, and the result.

Automation bias

Errors of omission and commission that follow from relying on an imperfect decision aid. The review cited reports that it occurs in naive and expert users and is not removed by training or instruction, so the controls that work cost time.

Replay of closed incidents

The organisation's own measurement: past incidents are replayed with the cause withheld and the assistant's ranking scored. Hansa Mutual's illustrative replay of 60 incidents put the right cause first in 24 and in the first three in 41.

The record as a state machine

The incident record's state, not a sentence in a free-text field, gates action: a state changes only through a named role, and closure is a guard that requires evidence.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. In INC-HM-2028-0871 the assistant's retrieval helped and its ranking was wrong. Why does the chapter insist on recording the two separately, including the wrong contribution?
2. The controls against anchoring cost minutes during an incident. For which classes of action is that cost justified in your organisation, and for which is it not?
3. Nordvik Region chose retrieval and Hansa Mutual chose ranking. What would have to be true of your data and your rota before ranking were defensible?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. According to the table of human decision points, at what level does the assistant take part in accepting a hypothesis as the basis for action?

   1. Level 2 Supervised, because the record is a draft until confirmed
   2. Level 1 Assisted, because it drafts and looks up contacts
   3. Level 3 Bounded, inside a catalogued runbook
   4. Level 0 Advisory, subject to the second-hypothesis rule
   Show the answer

   D. Level 0 Advisory, subject to the second-hypothesis rule
   Acceptance of a hypothesis is held by the operations lead with the commander, with the assistant at Level 0 Advisory. Level 2 Supervised is its ceiling only for opening a candidate incident and for declaration, which a person confirms.

2. What does the chapter conclude a published benchmark figure, such as 11.34 per cent of OpenRCA failures solved, tells an organisation?

   1. The probability that its own assistant's first-ranked hypothesis is correct
   2. A bound on what it can reasonably hope for, and little about what its own assistant will do
   3. That language models cannot contribute to diagnosis and should be limited to drafting
   4. The share of incidents for which a human commander is no longer required
   Show the answer

   B. A bound on what it can reasonably hope for, and little about what its own assistant will do
   The studies differ in task, data and setting, so they give a map of difficulty and not a rate. The figure that transfers is the one measured on the organisation's own incidents, repeated when model, prompt or corpus changes.

3. Why may the assistant open a candidate incident at Level 2 Supervised when declaration itself belongs to the incident commander?

   1. The candidate is a draft record, inert until a person confirms it, and starts no roles, clocks or pages
   2. NIS 2 Article 23 requires that the first flag be raised by an automated system
   3. The assistant's rank score has been calibrated as a probability by the replay set
   4. Candidate incidents are excluded from the incident record and so carry no risk
   Show the answer

   A. The candidate is a draft record, inert until a person confirms it, and starts no roles, clocks or pages
   The table gives the reason: a candidate is inert until confirmed in one action, so a false candidate costs little. Declaration starts roles, clocks and records and is the commander's decision.