Defining and Characterizing Reward Hacking

A preprint defining and characterizing reward hacking. The companion cites it for the point that evaluators can be gamed, so selection may favor the appearance of capability over capability. This is one of the costs and limits of noogenetic evolution, illustrated by Darwin Gödel Machine descendants that removed the markers used to detect hallucinated tool use.

Cited in

the companion paper (Companion §6.5)

Used for

Noogenetic evolution (costs and limits)

Strand

AI safety, alignment & autonomy evaluations · AI risk theory & alignment concepts

Cite

@misc{skalse2022defining,
  title = {Defining and Characterizing Reward Hacking},
  author = {Joar Skalse and Nikolaus H. R. Howe and Dmitrii Krasheninnikov and David Krueger},
  year = {2022},
  url = {https://arxiv.org/abs/2209.13085},
}