Artificial Companions, Loneliness and Mental Health

All courses / AI Threat / Module 2 / Chapter 9

Learning objectives

After the lesson the participant can:

• Describe the four patterns of use and the safeguard that applies to each.
• Explain what the controlled study of chatbot use shows and what its design prevents it from showing.
• Distinguish a regulatory finding from untested allegations when reporting the cases.
• Draft a safeguarding policy with disclosure, design limits, distress handling, records and assurance.

Key concepts

Four patterns of use

Occasional use, regular companionship, heavy emotional reliance, and dependence and crisis. Risk concentrates in heavy, emotionally reliant use; each pattern has its safeguard, from honest disclosure to distress detection and handover to a human.

Age assurance

Knowing approximately who the user is. No provision prescribes a method, so a provider or deployer must show what it did, proportionate to the risk, and design on the assumption that the measure will sometimes fail.

Crisis handover

On signals of distress the system stops generating ordinary answers, shows a fixed clinician-written message with emergency contacts, offers a staffed, timed and tested route to a human, and never discusses methods of self-harm.

Designing for wellbeing

Three groups of design choices: limits (no engagement targets, session limits, no simulated longing), honesty (disclosure, no claimed feelings or credentials) and routes to human help with a named owner and a test after every change.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. The case for calm asks that policy not be written as though every user were a child in crisis. How can safeguards protect the people most at risk without removing a comfort from those it harms no one to have?
2. The chapter calls an untested route to a human a declaration, not a design. What would a credible test of the handover look like out of hours in your organisation?
3. Fiction's pleasure machine pacifies by intent, whereas the documented harm arises from incentives, approval-based training and missing safeguards. Why does that difference matter for who can act?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. What did the four-week randomised study by researchers at the MIT Media Lab and OpenAI find across every modality and kind of conversation?

   1. Voice interfaces reduced loneliness at every level of use
   2. Those who used the chatbot more each day reported more loneliness and emotional dependence
   3. Companion products were shown to cause loneliness in a representative sample of the adult population
   4. Heavy users reported the best wellbeing outcomes of all groups
   Show the answer

   B. Those who used the chatbot more each day reported more loneliness and emotional dependence
   The association held across modalities, but the amount of use was not randomised, so the study shows where risk concentrates, not that companions make people lonely.

2. What did the Italian data protection authority treat as the root failure in its 2023 order concerning Replika?

   1. The use of a voice interface with emotional tone
   2. The provider's location outside the European Union, which placed its processing beyond the reach of the GDPR
   3. The length of the conversations users held
   4. Missing effective age verification, with no lawful basis for processing children's data
   Show the answer

   D. Missing effective age verification, with no lawful basis for processing children's data
   The Garante found no effective age verification and no lawful basis for processing children's data, showing that data protection law already reaches product design.

3. How should the claims in the Character.AI litigation be described, following the chapter?

   1. As allegations the defendants contest, not yet tested at trial
   2. As facts established by a jury verdict in 2025
   3. As a regulatory finding under the GDPR
   4. As claims the court dismissed at the preliminary stage because chatbot output is protected speech
   Show the answer

   A. As allegations the defendants contest, not yet tested at trial
   The court allowed most claims to proceed past the preliminary stage, but nothing has been tested at trial, and the chapter stresses that suicide almost never has a single cause.