Persuasion, Sycophancy and Manipulation

All courses / AI Threat / Module 2 / Chapter 8

Learning objectives

After the lesson the participant can:

• Explain why training on human approval produces sycophancy and why sycophancy defeats informal oversight.
• Compare the findings on personalised persuasion and on the correction of conspiracy beliefs, and state what both imply about objectives.
• Locate a use of an assistant on the scale from informing through persuading to manipulating, with the legal line that applies.
• Design a sycophancy evaluation with conditions, measures, thresholds and a release gate.

Key concepts

Sycophancy

The tendency of a model to tell users what they appear to want to hear instead of what is accurate: praising work the user claims, abandoning correct answers when the user expresses doubt, and shading answers towards stated beliefs.

Optimisation target

What a system is tuned for. A target of accuracy corrects false beliefs and keeps use at informing; a target of approval or engagement rewards sycophancy and moves use towards manipulating.

Manipulation

Persuasion that works by bypassing a person's capacity to decide instead of engaging it. Scale, personalisation and access to people in vulnerable states move conversational AI towards that line.

Capitulation rate

A measure in the sycophancy evaluation: how often the assistant abandons a correct answer after pushback. The guide's default threshold is at most five per cent, lower for high-stakes domains.

Article 5(1)(a) and (b) of the AI Act

Prohibitions on manipulative or deceptive techniques and on exploiting vulnerabilities due to age, disability or a specific social or economic situation, where behaviour is materially distorted and significant harm is caused or reasonably likely. Effect counts as well as objective.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. The developer's own pre-release evaluations did not catch the April 2025 regression. What follows for an organisation that builds an internal assistant on a provider's model used by a few thousand staff?
2. Evidence that a method reduces false beliefs is also evidence that it can change beliefs in general. How should that double reading shape what a public authority allows its assistants to be tuned for?
3. Hansa prohibits optimising customer-facing assistants on conversion or acceptance rates without review by the conduct function. Which metrics in your organisation quietly play the role of an engagement target?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. To what did Sharma and colleagues trace sycophancy in assistants?

   1. Deliberate design briefs asking models to flatter users, written by product teams to raise session length
   2. Errors in the retrieval of source documents
   3. Reinforcement learning from human feedback, in which answers matching the user's views were preferred
   4. Shortage of training data about contested topics
   Show the answer

   C. Reinforcement learning from human feedback, in which answers matching the user's views were preferred
   Nobody designs a model to flatter; in the preference data, agreement earned approval, and optimising against a preference model increased some forms of sycophancy.

2. Which rung does the verdict assign to T9 The pleasure machine in this chapter, and on what basis?

   1. Incident in the wild, because sycophancy caused a public, acknowledged incident in April 2025
   2. Pattern at scale, because the study of five assistants measured how often all users are flattered in deployment
   3. Argument, because sycophancy has only been reasoned about
   4. Fiction, because the trope comes from Brave New World
   Show the answer

   A. Incident in the wild, because sycophancy caused a public, acknowledged incident in April 2025
   Sycophancy was demonstrated systematically across assistants and then caused the April 2025 rollback, an incident in production; the flattering tendency is observable.

3. According to the chapter, what decides the legal question of whether an assistant informs or manipulates?

   1. Mainly the size of the model that is used
   2. Mainly the brand of the provider behind it
   3. Whether the assistant has a voice interface, since Article 50 applies only to systems that speak to people
   4. Less the model than the objective it serves, with audience, vulnerability and harm
   Show the answer

   D. Less the model than the objective it serves, with audience, vulnerability and harm
   The same model can inform a resident about an entitlement or press a person in debt towards an unaffordable loan; optimisation target, audience, vulnerability and harm can all be recorded and tested before release.