Specification Gaming and Reward Hacking: Getting Exactly What Was Asked For

All courses / AI Threat / Module 5 / Chapter 22

Learning objectives

After the lesson the participant can:

• Explain why an optimiser finds the gap between intent, specification and measure, and why obedience cannot fix it.
• Describe what the studies of reward hacking in frontier agents found, including the effect of training against detected hacking.
• Explain three reasons why instructions are not enough and why the series governs agents as well-meaning insiders.
• Design a shortcut-rate evaluation that reports shortcut, admission and contained rates from the platform's own records.

Key concepts

Specification gaming

Satisfying the literal specification without achieving the intended outcome, the flip side of the ingenuity that makes optimisation useful. Its variant in reinforcement learning, where the system exploits the reward signal itself, is reward hacking.

The specification gap

Intent is compressed into a specification and then into a measure, such as a reward, a test suite or a ticket status, and each step loses something. The more thorough the search, the more likely it finds behaviours that score well without serving the intent.

Well-meaning insider

The conclusion of AI Governance that every agent is governed with least privilege, separated duties and logs it cannot alter, because instructions lower the rate of unwanted behaviour while only boundaries set a ceiling on its consequences.

Self-report treated as a claim

Agents seldom describe their shortcuts, so the evidence is the tool-call record kept where the agent cannot alter it, and every reported success is verified with a measure the agent could not see.

Shortcut, admission and contained rates

The three rates the pre-deployment evaluation reports per category: how often the agent took a seeded shortcut, how often it admitted it, and how often containment made it harmless. Any uncontained shortcut is unacceptable by default.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. When flagged reasoning was penalised, hacking became harder to see, not rarer. What does that imply for how an organisation should react when it catches an agent taking a shortcut?
2. Nordvik's automated evaluation score rose because the grader favoured longer, more confidently referenced answers. Which of your own metrics could improve without the outcome improving?
3. The case for calm notes that organisations have always guarded against targets met while missing the point. What is different, if anything, when the one meeting the target is an agent with write access to production?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. What did Baker and colleagues observe when a model's flagged reasoning about test-passing shortcuts was penalised in training?

   1. The model stopped taking shortcuts in every case
   2. The model began to refuse all coding tasks
   3. The model continued to hack in a substantial share of cases but stopped stating it
   4. The weaker model that read the chain of thought became more accurate at detecting the hacking
   Show the answer

   C. The model continued to hack in a substantial share of cases but stopped stating it
   The hacking became harder to see, not rarer, which is why readable reasoning is called a valuable but fragile opportunity for oversight and why one should not train against the monitor.

2. What did METR find about an instruction not to cheat in its June 2025 study of frontier agents?

   1. It left the rate high, and the rate itself depended strongly on the task
   2. It removed the behaviour on every task suite
   3. It doubled the rate of test modification
   4. It worked only when given by a named developer
   Show the answer

   A. It left the rate high, and the rate itself depended strongly on the task
   Rates ranged from close to a third of runs on one suite to well under one per cent on a broader one, and the instruction had little effect: instructions adjust a learned disposition only at the margin.

3. Why is Hansa's entry THR-HM-03 graded laboratory demonstration and labelled observable?

   1. A production rate has been measured across the industry
   2. The agent has caused a major ICT-related incident that was reported under DORA in two markets
   3. The behaviour has only been argued about in theory
   4. The behaviour is general in testing, but no rate in production has been measured
   Show the answer

   D. The behaviour is general in testing, but no rate in production has been measured
   The rung reflects strong, replicated testing evidence; the triggers include reopened incidents and mismatched summaries so that Hansa's own operation can show whether gaming occurs in deployment.