Observe: Telemetry, Detection and Response Compared with Expectation

All courses / AI Purple / Module 2 / Chapter 9

Learning objectives

After the lesson the participant can:

• Explain why the expectation is written from the detection catalogue and sealed before the ground truth is unsealed.
• Classify emulated steps into the seven outcomes, recording the earliest failing layer.
• Draft a finding that states the expectation, the outcome, the layer that failed and the evidence, without choosing the fix.
• Review the use of an AI correlator at P3 against its four failure modes and the privacy rule.

Key concepts

Expectation sheet

The organisation's coverage claim made explicit for the techniques in a scenario: telemetry, rule, severity, triage and time bound per step. The detection owner writes it from the catalogue, knowing only the list of techniques, and the white team seals it.

Ground truth

The record of what the emulation did, written by the emulation side at the time. It is tagged, time-stamped from a source both sides trust and held outside the system observed, and the tag must never reach detection logic.

Seven outcomes

Prevented; detected; alerted but not triaged; triaged but not acted on; missed; no telemetry; not tested. They are mutually exclusive and ordered by layer, and detected demands the whole chain through a qualified triage decision and the playbook action.

Purple session record

The record (PRP) of the meeting in which red and blue read the result together: both seals, the roles, one outcome per step with a saved query, disagreements recorded with both readings, and results reported by technique, never by analyst.

Optimistic join

A correlator's failure in which it finds an alert near in time and on a related host and counts the step as detected. At Hansa Mutual all three corrections to the correlator's proposals were of this kind.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. Step 2 at Nordvik Region looked detected in the SIEM alone. Which of your own detected results would survive the question of whether a qualified person took a triage decision in time?
2. Unsettled steps take the more cautious outcome, and disagreement is not settled by seniority. What does that rule cost a session, and what does it protect?
3. A Mode 5 agent that runs a scenario does not also grade it. Where else in your tooling does one system both act and judge its own result?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. A detection fired and an alert was closed by an automated process without a qualified triage decision. Which outcome does the chapter record?

   1. Detected, with a note that automation handled the alert.
   2. Missed, because no person ever saw the alert.
   3. Alerted but not triaged, with the reason closed by automation.
   4. Triaged but not acted on, because closure counts as triage.
   Show the answer

   C. Alerted but not triaged, with the reason closed by automation.
   Detected demands the whole chain: detection, alert, a qualified triage decision and the playbook action within the bound. An alert closed by automation without a qualified decision is alerted but not triaged, with the reason recorded.

2. What does the detection owner receive before writing the expectation sheet?

   1. The list of techniques in the scenario, and nothing of the plan, order or timing.
   2. The full emulation plan, so that the expectations match each step.
   3. The emulation log, so that expectations can be checked against it.
   4. The tester's draft report, with the findings removed by the white team.
   Show the answer

   A. The list of techniques in the scenario, and nothing of the plan, order or timing.
   The sheet is written from the detection catalogue, and the owner needs only the list of techniques, not the plan, the order or the timing. Reading the log first would let the tested side grade its own test.

3. Why does the chapter record the layer at which telemetry disappeared for a no telemetry outcome?

   1. Because the layer sets the severity rating given by the tester.
   2. Because supervisors require the layer in the attestation.
   3. Because the correlator cannot join steps without a layer field.
   4. Because the layer decides who owns the fix.
   Show the answer

   D. Because the layer decides who owns the fix.
   An event may be not generated, not forwarded, dropped or retained too briefly. Each has a different owner, as Hansa Mutual's three telemetry gaps showed, so the observer records the layer at which the event disappeared.