The Pressure on the Loop: Machine-Speed Adversaries, Detection Decay and the Annual Test

All courses / AI Purple / Module 1 / Chapter 3

Learning objectives

After the lesson the participant can:

• Explain the assumptions of the annual rhythm and the evidence the chapter cites against each.
• Calculate a test-to-change ratio for a priority technique from change and evidence records.
• Classify a detection as current, at-risk or expired from its last passing test and its dependency changes.
• Compare what AI makes cheaper in the loop with what stays costly.

Key concepts

Test-to-change ratio

For a given technique, the number of production changes that touch what the technique depends on, between two passing tests. A ratio of one means every change was followed by a verification; a ratio of twenty means twenty changes accumulated before anyone looked.

Detection decay

The loss of a detection's effect between tests as its dependencies change. It is measured by age since the last passing test, set against the evidence expiry, together with the count of dependency changes since that test.

Current, at-risk, expired

The three states of a decay estimate: current if a test passed within expiry and no dependency changed since; at-risk if within expiry but a dependency changed; expired if the last passing test is older than the expiry.

Finding half-life

The time a finding spends between P3 Observe and P5 Verify, in two parts: the wait for a change at P4 Improve and the wait for the proof of that change.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. Age alone misleads in both directions. Which of your detections would look healthy by age and unhealthy by dependency changes, or the reverse?
2. The chapter says a calendar set by regulation or procurement rewards a report. What would have to change in a contract or a testing programme for it to reward a verified finding instead?
3. If AI makes variation and repetition cheaper but not novelty, judgement, independence or safety, where does the bottleneck of your own loop move?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. A priority detection at Hansa Mutual passed its last test 53 days ago, and its log connector was upgraded after that test. Which state does the chapter's decay estimate give it?

   1. Current, because 53 days is inside the 90-day expiry.
   2. At-risk, because it is within expiry but a dependency has changed.
   3. Expired, because any dependency change voids the evidence.
   4. Retired, because the connector upgrade replaced the detection.
   Show the answer

   B. At-risk, because it is within expiry but a dependency has changed.
   A detection is at-risk when its last passing test is within the evidence expiry, 90 days for a priority technique at Hansa Mutual, but a dependency has changed since. Expired applies only when the last passing test is older than the expiry.

2. What does a test-to-change ratio of twenty mean for a technique?

   1. Twenty passing tests were run for each production change on the technique's dependency path.
   2. Twenty analysts would be needed to verify the technique.
   3. Twenty changes accumulated on its dependency path before anyone verified it.
   4. Twenty per cent of the scenario library covers the technique.
   Show the answer

   C. Twenty changes accumulated on its dependency path before anyone verified it.
   The ratio counts production changes touching what the technique depends on between two passing tests. At twenty, twenty changes accumulated, and if the twentieth broke something the test cannot say which of them did.

3. According to the chapter, which of the following does AI not make cheaper in the loop?

   1. Independence between the side that tests and the side that is tested.
   2. Repetition of a catalogued, reversible scenario on a schedule.
   3. Variation of a scenario's parameters such as account and hour.
   4. Drafting of scenario variants and of result summaries.
   Show the answer

   A. Independence between the side that tests and the side that is tested.
   Repetition is already cheap and variation and drafting are expected to become cheaper. Novelty, judgement, independence and safety stay costly, and a design that assumes otherwise moves the bottleneck without removing it.