The End of Truth? Synthetic Content and the Information Ecosystem

All courses / AI Threat / Module 3 / Chapter 10

Learning objectives

After the lesson the participant can:

• Explain the liar's dividend and why provenance, not labelling, is the defence against it.
• Split T10 into its parts and grade each on the evidence ladder as the verdict does.
• Describe the duties of providers and deployers under Article 50 of the AI Act and the role of recorded human review.
• Draft a content-provenance policy with a canonical channel, signing, corrections and a confirmation procedure.

Key concepts

The liar's dividend

Once the public knows that convincing audio and video can be fabricated, anyone caught on a genuine recording can claim it was fabricated, and some will believe the claim. The mechanism needs no fake, only its plausible possibility.

Slop

Low-quality synthetic content produced at scale, defined by indifference to truth. It is economic before it is technical: the marginal cost of plausible content is almost nothing while advertising and search still pay for attention.

Model collapse

The degenerative process found when successive models are trained on the previous generation's output: the tails of the distribution disappear first. The chapter grades it a laboratory demonstration of a mechanism under defined conditions, not a forecast.

Label versus provenance

A label says that content was generated or manipulated; provenance says where it came from and what happened to it since. Labels help against the fake that would be believed, provenance against the liar's dividend.

Canonical record

The official version of what an organisation says, on a channel it controls, with text, date, signed media and a hash of each file, so that a statement can prove its origin even when a platform strips credentials.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. The chapter argues that the bottleneck in the spread of falsehood was never production but whether people found a story worth sharing. If so, what exactly does cheap synthetic content change?
2. A summary needs the journalism and reduces the income that pays for it. Who in your country bears that cost first, and what follows for an organisation that depends on local or specialist reporting?
3. Hansa lets detector scores route a claim to specialist review but never refuse it on their own. Why is that rule important for a customer whose genuine photograph is wrongly flagged?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. What did the Pew measurement of browsing data find about Google searches that produced an AI summary?

   1. Users clicked on a source cited inside the summary in about half of visits, more often than on ordinary result links
   2. Users clicked on links more often than without a summary
   3. Users clicked a link to another website in about 8 per cent of visits, against about 15 per cent without one
   4. Users stopped using search engines for news altogether
   Show the answer

   C. Users clicked a link to another website in about 8 per cent of visits, against about 15 per cent without one
   Clicks to other websites roughly halved when a summary appeared, and clicks on a source cited inside the summary were about 1 per cent, which is the measured pressure on the economics of news.

2. According to the study of about 126,000 rumour cascades, what drove the faster spread of false news?

   1. Human behaviour, since automated accounts spread true and false news at about the same rate
   2. Automated accounts, which spread false news far more than true news across every category the study classified
   3. Generative AI, which produced most of the false stories
   4. Fact-checking organisations, which amplified the rumours
   Show the answer

   A. Human behaviour, since automated accounts spread true and false news at about the same rate
   The study covered 2006 to 2017, before generative AI, and found that people, not automated accounts, drove the difference; the authors suggested novelty explains part of the appeal.

3. How does the verdict treat the confirming evidence Chapter 2 named for T10, a measurable loss of trust or of the ability to establish facts attributable to AI?

   1. It has reached pattern at scale across Europe
   2. It was shown by the Romanian court decision of 2024
   3. It is an incident in the wild in every member state
   4. It has not been shown
   Show the answer

   D. It has not been shown
   The capability is established and fraud has reached incident in the wild, but the measurable loss of trust attributable to AI has not been shown; a society where nobody can tell what is real is speculation.