Data Architecture for AI

All courses / AI Platform and Security / Module 2 / Chapter 6

Learning objectives

After the lesson the participant can:

• Explain why the quality, legality and reproducibility of a model are bounded by its data.
• Draft a data contract with the two kinds of field that AI consumers add: distributions and permitted purposes.
• Distinguish a feature store from a vector store by content, consumer and typical failure.
• Compare the lawful basis for model training available to a private organisation and to a public authority.

Key concepts

Data product descriptor

A file the platform can read that names the product's owner, outputs, classification, lawful basis, retention, datasheet and lineage. Its lawful basis block separates operational use from model training because the GDPR does.

Data contract

A statement of what the consumer may rely on, in a form a test can check. The platform verifies it on every publication, so that when the contract breaks the producer's pipeline fails, not the consumer's model.

Lineage and provenance

Lineage answers where a value came from; provenance answers how an artefact came to exist. The platform captures lineage instead of teams declaring it, so that it can say which models consumed a dataset version.

Point-in-time join

The defining capability of a feature store, which keeps future information out of a training set. A vector store's problem is different: access control, since a chunk is still its source document.

Model collapse

The predictable degradation of models trained recursively on data generated by earlier models: first the tails of the distribution disappear, later the output converges on a few high-probability patterns. Synthetic data is therefore labelled in lineage.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. The chapter says purpose limitation is the GDPR principle AI programmes break most often. Which dataset in your organisation was collected to run a service and is now a candidate for training?
2. RET-04 does not promise to delete a person from a trained model at once. How would you explain that position to a data subject and to an auditor?
3. The known gaps section of a datasheet is said to be the one that changes model design. What would an honest known gaps section say about a dataset you rely on?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. Which two kinds of field does a data contract add for AI consumers, according to the chapter?

   1. The name of the model team and the size of the training cluster
   2. The distribution of numeric columns and the permitted purposes
   3. The price per row and the licence of the pipeline framework
   4. The embedding model version and the chunking strategy
   Show the answer

   B. The distribution of numeric columns and the permitted purposes
   A model served on a different distribution has silently changed behaviour, and data collected under one lawful basis cannot be repurposed by the consumer.

2. Why can a public authority not rely on legitimate interest for training a model in the performance of its tasks?

   1. Because the Data Governance Act reserves legitimate interest for data intermediation services under its Chapter III
   2. Because EDPB Opinion 28/2024 rejects legitimate interest as a lawful basis for developing or deploying any model
   3. Because Article 9 of the GDPR prohibits public bodies from processing personal data
   4. Because the final sentence of Article 6(1) of the GDPR excludes it, leaving public task under Article 6(1)(e)
   Show the answer

   D. Because the final sentence of Article 6(1) of the GDPR excludes it, leaving public task under Article 6(1)(e)
   The authority relies on public task laid down in law, so its descriptor points to the authorising provision, and a dataset without one cannot be used for training.

3. How does the vector store handle access control in the chapter's design?

   1. The index carries its sources' access attributes, and the retriever filters on entitlements before similarity search
   2. The generator is instructed in the system prompt not to quote passages the caller may not read, and a canary string detects leaks
   3. Restricted documents are embedded with a separate model that only administrators may call
   4. The grounding check removes restricted passages after the answer has been generated
   Show the answer

   A. The index carries its sources' access attributes, and the retriever filters on entitlements before similarity search
   A chunk is still its source document, so entitlements are applied before search and no restricted chunk becomes a candidate.