Surveillance, Profiling and the Erosion of Privacy

All courses / AI Threat / Module 3 / Chapter 13

Learning objectives

After the lesson the participant can:

Explain the three ways in which AI extends the reach of surveillance: analysis of unstructured data, inference and cheaper linkage.
Classify uses that observe or judge people as permitted under conditions, high-risk or prohibited under Article 5 of the AI Act.
Describe how a lawful ranking model can drift into social scoring through a single feature change.
Design a prohibited-practice check that runs whenever an AI inventory entry changes.

Key concepts

Linkage: Combining datasets that may each have been collected lawfully into a profile no single purpose justified. SyRI was in essence a linkage machine, and cheaper linkage is done more often.
Social scoring: Under Article 5(1)(c): evaluating or classifying people over time on social behaviour or personal characteristics, where the score leads to detrimental treatment in unrelated contexts or out of proportion to the behaviour. Scoring as such is not prohibited.
Political filtering in models: A developer's values, and those of its state, travel with the weights. The chapter adds political filtering to the origin assessment of AI Governance, tested against a reference record whenever the model changes.
Fails closed: A property of the prohibited-practice check: an inventory entry that does not declare the attributes the rules read cannot be merged. The check does not decide legality; it makes the questions unavoidable when a change is proposed.
Representation agreement as design input: A works-council or union agreement states purpose, data, retention and who may look at individual data: the controls a well-designed SOC needs anyway, such as pseudonymisation until an investigation is opened.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Interactive exercise 1 · Sort

Permitted, high-risk or prohibited

Sort each use of AI that observes or judges people into the class the chapter's table gives it under the AI Act.

Interactive exercise 2 · Scenario

Rolling out the SOC investigation agent

Apply the chapter's treatment of employee-representation rights as a design input.

Teaching plan

12 min	The teacher presents the two faces of the watching machine and what AI adds in reach; participants name one linkage in their organisation that no single purpose would justify.
10 min	Review the Clearview decision and the studies of political filtering, separating what was measured from who measured it and under what circumstances.
15 min	Walk through the Article 5 prohibitions on surveillance and profiling and Table 13.1; participants sort ten example uses into the three classes.
28 min	Exercise in groups: write a prohibited-practice check for the AI inventory modelled on PPC-NR-01 and run it on two proposed changes, one of them a new feature from an unrelated context.
10 min	Discuss the employee and the SOC, stage the debate on whether the safeguards will hold, and review the verdict.

Discussion

In Nordvik's example run nobody intended social scoring; an analyst noticed that travel-card activity improved the ranking. What in your change process would catch a well-meant improvement of this kind?
The case for concern says infrastructure persists and is repurposed. Which data stores or monitoring tools in your organisation would be most valuable to a future owner with different intentions?
The difference between defending the organisation and monitoring its people lies in purpose limits, access rules and oversight, not in the technology. How would an employee of yours verify that the limits are kept?

Written exercise

Exercise · 28 minutes

A prohibited-practice check for the AI inventory

Write a prohibited-practice check for your organisation's AI inventory. State what triggers it: a new entry, a change to features or data sources, a change of purpose. Write one rule for each of four practices: social scoring, untargeted scraping for facial-recognition databases, emotion recognition at work or in education, and real-time remote biometric identification. For each rule list the declared attributes it reads, the conditions under which it applies and the outcome: block, block unless an exception is approved, or legal review before merge. Then run the check by hand on two realistic changes to systems you know and record the rule hit, the reason and the outcome.

Deliverable: A check with triggers, four rules and outcomes, plus two example runs recorded with rule hit, reason and outcome.

Your work

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

What did the French data protection authority's 2022 decision against Clearview AI establish, as the chapter reads it?
Show the answer
A. Publicly accessible is not freely usable: there was no legal basis for processing the biometric data
The CNIL imposed a penalty of 20 million euros, found no legal basis under the GDPR and failures to respect access and erasure rights, and ordered deletion; Article 5(1)(e) of the AI Act later prohibited such databases.
Under Article 5(1)(f) as the chapter describes it, which use is prohibited?
Show the answer
C. Inferring a worker's stress from voice or face in the workplace, absent a medical or safety reason
The prohibition covers inferring emotions from biometric data at work and in education. Sentiment analysis of text is outside it, and emotion recognition elsewhere is high-risk, not prohibited.
Why does the chapter say security AI belongs in the same inventory pipeline as every other system?
Show the answer
B. A SOC tool that evaluates workers is high-risk, and one inferring stress from voices would be prohibited
The purpose of security monitoring is legitimate, but the same tools can become prohibited or high-risk through use, so they pass the same check and the employee-representation process.