Detect: Detection Engineering as Code

All courses / AI Blue / Module 2 / Chapter 6

Learning objectives

After the lesson the participant can:

• List the six things a detection record holds that a rule in a console does not
• Review an AI-drafted detection against the six steps of the chapter's protocol
• Distinguish unit, replay, simulation and purple-loop tests by what each shows
• Classify detection debt as stale, untested, noisy, blind or orphaned and propose a way to pay it down

Key concepts

Detection record

The versioned definition of one detection, identified as DET-NR-NNNN or DET-HM-NNNN. It holds logic, data sources with a health check, an ATT&CK mapping, tests, ownership and cost, and origin.

Test gate

Automated checks on the pull request: a schema check, a logic check that every field exists in the data source's schema, and unit tests for positive and negative cases. It sits before human review.

Alert budget

The number of alerts per week the SOC agrees to take from a detection. The default for a new medium-severity detection is no more than six alerts a week for the whole SOC.

Shadow rollout

The first stage of rollout, in which a detection writes alerts to a log that no analyst reads, so that its volume and precision can be measured against real traffic before a pilot and full release.

Detection debt

The accumulated cost of detections that no longer do what they claim. Five kinds recur: stale, untested, noisy, blind and orphaned. The register that lists them is generated from the records, not kept by hand.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. The chapter says a test suite written entirely by the system that drafted the logic proves little. What would count as an independent test in your pipeline?
2. A repository history shows who wrote and approved each change. How should a team use that history without turning it into a measure of individuals?
3. The book rests its refusal of Level 4 for detection content on the cost of a silent miss, not on irreversibility. Why is a silent miss harder to notice than a wrong containment?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. What is the chapter's central discipline for accepting a detection?

   1. It is accepted when the supplier certifies its coverage of the ATT&CK matrix
   2. It is accepted when a test shows that it fires and a measure shows what it costs in false positives
   3. It is accepted when the drafting assistant reports high confidence in the logic
   4. It is accepted when two analysts agree that the rule looks reasonable
   Show the answer

   B. It is accepted when a test shows that it fires and a measure shows what it costs in false positives
   The chapter states that a detection is not accepted until a test shows that it fires and a measure shows what it costs. A detection counts only when its test has run recently.

2. Which kind of detection debt describes a detection whose data source is silent or lacks a required field?

   1. Stale
   2. Orphaned
   3. Noisy
   4. Blind
   Show the answer

   D. Blind
   Blind debt means the detection cannot fire because its data source is silent or lacks a field. Stale means the review date has passed, orphaned that the owner has left, noisy that alerts exceed the budget.

3. What status do supplier-held detections have until the contract grants provenance, logic access, tuning rights and test rights?

   1. They are treated as Mode 0 Ambient AI and stay declared rather than verified
   2. They are counted as verified in the coverage map, because the supplier updates them continuously
   3. They are delegated at Level 3 Bounded under the supplier's playbook
   4. They are removed from the coverage map and disabled in the console
   Show the answer

   A. They are treated as Mode 0 Ambient AI and stay declared rather than verified
   The summary states that supplier detections are Mode 0 Ambient AI until the contract grants the four rights, and until then they stay declared.