Model Lifecycle Management: MLOps and LLMOps as Code

All courses / AI Platform and Security / Module 2 / Chapter 7

Learning objectives

After the lesson the participant can:

• List the six promotion gates and say what each checks.
• Compare shadow, canary and champion-challenger deployment by what each measures.
• Explain how LLMOps changes the versioned artefact, the evaluation set, the metric and the promotion unit.
• Draft a predetermined change envelope that separates planned retraining from changes needing a new assessment.

Key concepts

Promotion gate

A set of automated checks a model version must pass to move between stages. The book insists on six: data, reproducibility, evaluation, comparison, readiness and approval. Thresholds are read from the decision record so that record and gate cannot disagree.

Training manifest

The file stored with the model that pins code by commit, data by immutable version, environment by image digest, configuration including seeds, and hardware class, with a declared reproducibility tolerance.

ML Test Score

A rubric of twenty-eight tests in four areas. A test scores half a point when manual and documented and a full point when automated, and the overall score is the minimum across the areas.

LLMOps

The lifecycle practice for a general-purpose AI model obtained from a third party: it versions prompt templates, retrieval and guardrail configuration and adapters, and promotes the compound system of model plus prompt plus configuration.

Predetermined change envelope

The retraining procedure, thresholds and acceptable ranges of change declared up front under Article 43(4) of the AI Act, so that each retraining within it is a planned change through the gates, with a list of what falls outside.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. Only the approval gate involves a person. Which of the other five gates is still performed by hand in your organisation, and what would automating it require?
2. The release note for claims-ranker 4.0.0 reports a small fall in precision alongside an improvement in parity. Who in your organisation would need such a note, and who writes it today?
3. A vendor model update is treated as a change to Hansa Mutual's own system. How do you learn of vendor updates today, and what do they trigger?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. How is the overall ML Test Score calculated from the four areas?

   1. It is the sum of the four area scores
   2. It is the average of the four area scores
   3. It is the minimum across the four areas
   4. It is the score of the monitoring area alone
   Show the answer

   C. It is the minimum across the four areas
   Taking the minimum means that exhaustive data tests cannot compensate for untested monitoring.

2. Why does the chapter call rollback a registry operation, and what does it require?

   1. Moving the champion alias back is immediate, provided the previous version's environment, features and dependencies still exist
   2. The registry retrains the previous version from its manifest whenever the alias is moved back
   3. The registry deletes the failed version, after which serving falls back to arrival order
   4. The registry asks the provider to redeploy the previous version within the change envelope
   Show the answer

   A. Moving the champion alias back is immediate, provided the previous version's environment, features and dependencies still exist
   Rollback works only while previous versions stay deployable, which is why at least two previous champions are kept warm for any high-risk system.

3. In Nordvik Region's change envelope, which of these is listed as outside the envelope?

   1. Retraining after 90 days on a newer release of the same data product
   2. A maximum tree depth of eight within the declared range
   3. A parity difference of 0.02 after retraining
   4. New or removed features, or a change of algorithm family
   Show the answer

   D. New or removed features, or a change of algorithm family
   Adding a feature is the change teams are most tempted to treat as routine; listing it means the pipeline requires a new decision record and assessment.