Evaluation and Testing of AI Systems

All courses / AI Platform and Security / Module 5 / Chapter 20

Learning objectives

After the lesson the participant can:

• Describe the six layers of the evaluation pyramid and how often each runs.
• Identify the four ways a golden dataset is mishandled and the governance that prevents each.
• Explain the documented biases of judge models and the rules that follow for using one.
• Draft an evaluation suite definition with datasets, checks, thresholds and reporting.

Key concepts

Evaluation pyramid

Six layers from numerous, cheap, deterministic tests run on every commit to few, costly ones run per release: unit tests for code, data tests, model metrics, behavioural and scenario tests, red teaming and human evaluation.

Golden dataset

A curated, versioned collection of inputs with expected outputs or labels, the reference for model metrics and scenario tests. It is mishandled through contamination, staleness, unclear ownership and personal data without proper controls.

LLM-as-a-judge

Using a strong LLM to grade outputs that have no reference answer. Judge models show position, verbosity and self-enhancement bias, so the judge is calibrated against a human-labelled sample, pinned, never of the same family as the system, and kept under a human layer.

Evaluation suite definition

The file that turns the pyramid into a pipeline gate. It declares datasets, metrics, thresholds and judge configuration; the pipeline fails if any threshold is not met and attaches the report to the evidence manifest.

Pass rate with confidence interval

Because a single pass or fail is not meaningful for a non-deterministic system, each scenario is run several times and the result compared with the champion, with a minimum improvement or a maximum permitted regression.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. A model that gains two points of accuracy and loses ten points of calibration has not improved for decision support. Which single headline number does your organisation report, and what does it hide?
2. Changes to expected outputs in an eligibility golden set are reviewed by the business owner, not by engineers. Who owns the expected answers in your evaluation sets?
3. Nordvik Region's red-team plan separates what is tested from how, and includes a caseworker among the testers. What does each of those choices make possible?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. Which rule about thresholds does the chapter set for changes to the evaluation suite?

   1. A threshold may be lowered whenever the challenger narrowly misses it
   2. A threshold is never lowered in the same pull request that changes the model
   3. Thresholds are set by the model provider and copied into the suite
   4. Thresholds apply only to high-risk AI systems and are optional elsewhere
   Show the answer

   B. A threshold is never lowered in the same pull request that changes the model
   Keeping the two changes apart means the decision to change the model and the decision to change the bar are made and recorded separately.

2. Which of these is one of the chapter's rules for using a judge model?

   1. Use the same model as the system under evaluation, so the styles match
   2. Prefer absolute scores over comparison between two versions
   3. Let the judge replace human evaluation once its agreement with human raters exceeds eighty per cent
   4. Calibrate the judge against a human-labelled sample and record the agreement rate as evidence
   Show the answer

   D. Calibrate the judge against a human-labelled sample and record the agreement rate as evidence
   The other rules are to prefer relative comparison, never use the same model or family as the system, pin the judge version, and keep a human layer above the judge.

3. In Nordvik Region's evaluation report the automated gate passed. Why did the release not proceed on that alone?

   1. Because the champion had scored higher than the challenger on routing accuracy and on explanation grounding
   2. Because the judge model had not been calibrated against human raters
   3. Because the gate is necessary but not sufficient: human evaluation and business approval were still open
   4. Because the injection resistance check had failed its minimum pass rate
   Show the answer

   C. Because the gate is necessary but not sufficient: human evaluation and business approval were still open
   The report lists an open item for human evaluation of sampled explanations and a pending business approval; the release proceeds only after both.