Measuring the Transformed Blue Team: Speed, Coverage and Quality of Judgement

All courses / AI Blue / Module 4 / Chapter 20

Learning objectives

After the lesson the participant can:

• Explain how each of the four traditional SOC measures is gamed and what it hides
• Select measures from the four families and attach a gaming check to each
• Draft a one-page board report with counts beside every figure and at most three decisions
• State the four limits on measures about people and apply the minimum group size rule

Key concepts

Verified incidents as the denominator

Speed is measured on verified incidents, not alerts. An alert closed as benign does not enter the denominator, and an incident found by a hunt, a purple exercise or a third party does.

Gaming check

The check paired with a measure that shows when it is being bought: reversal share for time to contain, the date of the last passing test for coverage, the seed protocol for judgement, a passing re-test for learning.

Cost per verified case

The cost of the SOC's analyst time, platform and agent budget divided by the number of cases verified as incidents. It prices judgement and replaces cost per alert.

Board page

One page with at most three decisions, each traced to a measure. Every figure is shown with its count, because 0.86 from 28 seeds and 0.86 from 280 seeds are different statements.

Minimum group size

A figure for a group so small that it identifies a person is an individual figure. Nordvik Region's six analysts are therefore reported as a whole unit, never by shift, week or reviewer.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. A measure is safe as a target only if the behaviour that improves it is the behaviour the organisation wants. Which of your current targets fails that test?
2. The practical guide says to set thresholds, not targets, and to express them as stop rules. What changes in behaviour when a number triggers a stop instead of a reward?
3. Hansa Mutual's problem is too much data and Nordvik Region's is that it cannot yet see its provider. Which is the harder position from which to report honestly to a board?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. What happens to mean time to respond and closure rate on the day an AI system starts closing defined classes of alert?

   1. They become impossible to compute from the case record
   2. They worsen until the analysts have learned the new tool
   3. They improve, whether or not the system closes correctly
   4. They stay the same, because closures by AI are not counted
   Show the answer

   C. They improve, whether or not the system closes correctly
   AI closure improves both measures the day it is switched on regardless of correctness, which is why the chapter says closure rate is safe as a target only if every closure is correct, the property it does not observe.

2. What is the gaming check for the coverage family?

   1. The number of rules deployed in the SIEM
   2. The date of the last passing test, carried in every cell of the map
   3. The share of alerts closed within the shift
   4. The supplier's statement of ATT&CK coverage for its default detections
   Show the answer

   B. The date of the last passing test, carried in every cell of the map
   Coverage can be inflated by declaring detections and never testing them or by mapping one rule to many techniques. The date of the last passing test in every cell exposes it.

3. Which rule does the chapter set for measures about people?

   1. Individual figures are shared only with the analyst's line manager
   2. The existence of seeding is kept secret from analysts so that nobody can prepare
   3. A ticket field collected for another purpose may be reused for appraisal once it is available
   4. No measure is computed per named analyst, and agreement with representatives precedes collection
   Show the answer

   D. No measure is computed per named analyst, and agreement with representatives precedes collection
   The four limits are no individual scoring, a minimum group size, agreement before collection, and seeds that are not traps for individuals: analysts are told that seeding exists.