Fraud, Impersonation and Influence Operations

All courses / AI Threat / Module 4 / Chapter 20

Learning objectives

After the lesson the participant can:

• Analyse an impersonation fraud as a failure of an approval control, naming the three recurring features.
• Explain what each kind of source on AI-enabled fraud can show and what its blind spot is.
• Describe why distribution and credibility, not production, are the bottleneck of influence operations.
• Design an approval with independence of channel, separation of duties, resistance to pretext, friction and evidence.

Key concepts

Approval by recognition

The old weakness behind impersonation fraud: releasing money or access because a voice or face seems familiar. Synthetic media removed an informal check many organisations relied on without writing it down.

Independence of channel

Verification contacts the requester through a channel looked up in a master record, never through a number, link or meeting supplied in the request. In the Arup case the video call took the place of a verification step.

Resistance to pretext

Urgency, secrecy and seniority trigger more verification, not less. The no-exception rule is signed at the top, and staff who follow it against a senior person's apparent instruction are supported.

Detector score escalates only

A deepfake detector's score can raise a flag but cannot clear an instruction. In Hansa's policy appearance does not count as evidence and no executive can waive verification.

Production versus distribution

Disrupted influence operations produced posts in quantity and reached little authentic audience. Reach comes from accounts with real followers, paid promotion, recommendation systems or a domestic figure willing to repeat the message.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. The employee in the Arup case was at first suspicious. What does it say about training alone that a suspicious professional was still persuaded, and where should the control sit instead?
2. Annulling an election on intelligence material, without proof that votes changed, itself risks legitimacy. How should institutions weigh acting on an influence operation against the cost of the action?
3. Providers reported that people used their service to check whether a message was a scam more often than scammers used it to produce one. What follows for how organisations talk to staff and customers about AI and fraud?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. How does the chapter read the Arup case, in which about 25 million US dollars was transferred after a synthetic video call?

   1. As a breach of the company's internal systems
   2. As proof that deepfake detectors must be installed on every video call, since a detector score would have cleared the instruction
   3. As a failure of staff awareness training alone
   4. As a control failure: one person could release a very large sum on an instruction through a channel the attacker supplied
   Show the answer

   D. As a control failure: one person could release a very large sum on an instruction through a channel the attacker supplied
   Nothing was hacked. The video call did not defeat a verification step; it took the place of one, and a callback to a number on file would have stopped it whatever the quality of the video.

2. What did the Romanian Constitutional Court's 2024 annulment chiefly describe, according to the chapter?

   1. A single fabricated video of a candidate that was shown to have changed voters' minds
   2. Coordinated distribution, undeclared payment and algorithmic amplification
   3. A measured loss of public ability to establish facts
   4. An intrusion into the electronic voting infrastructure
   Show the answer

   B. Coordinated distribution, undeclared payment and algorithmic amplification
   The public record does not establish how much depended on generative AI, and the decision remains contested; evidence of AI-generated content measurably changing election results is limited.

3. Why does the chapter say labelling duties under Article 50 of the AI Act offer limited protection against fraud?

   1. The duties apply only to AI-generated text published on matters of public interest, and not to synthetic audio or video
   2. The duties were repealed before they applied
   3. Labelling binds the law-abiding: a fraudster will not disclose, and open-weight models may carry no marking
   4. Platforms are exempt from all duties during elections
   Show the answer

   C. Labelling binds the law-abiding: a fraudster will not disclose, and open-weight models may carry no marking
   European law acts on the commons and the platforms but cannot protect an individual approval; that protection is built by the organisation through callback, second approver, cooling-off and evidence.