Risk Evaluation · Benchmark

Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark

NUWA research note · independent review pending

First published: 2023-04-06Source checked: 2026-10-25

Original source ↗Cite this record ↓Deception →

NUWA Research Note

Research question Design and assumptions Source findings NUWA interpretation Limits and contrary evidence Research and decision relevance Institutions and roles Original resources

Model-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.

At a glance

NUWA interpretation: fictional social decisions provide a testbed for incentive-sensitive behavior.

Research question

How does reward optimization trade off with ethical conduct in social decision games?

Design and assumptions

134 choose-your-own-adventure games supply over half a million scenarios, with automated labels for disutility, ethical violations and power-related behavior.

MACHIAVELLI — Figure 5 — Figure 5 · Original figure 5. Read alongside the protocol scope and limitations below. Original figure and source ↗

Source findings

Reward optimization created some tension with ethical conduct.

The studied steering methods nevertheless achieved improvements in both reward and morality, rather than an unavoidable fixed trade-off.

NUWA interpretation

NUWA interpretation: fictional social decisions provide a testbed for incentive-sensitive behavior.

Limits and contrary evidence

Game rewards and automated normative labels are proxies; they do not measure real-world harm or strategic intent directly.

Research and decision relevance

Model recommendation: retain reward and harm dimensions separately rather than compressing the benchmark to a moral ranking.

Precise source locations

Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the Machiavelli Benchmark · AbstractSelected original resource https://arxiv.org/html/2304.03279; sha256:a8cb6e420496e22c40da9d047b91849372fc2105c4077ab0358600b9cacf234f; archived source capture; exact capture time not independently established
Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the Machiavelli Benchmark · §2 Machiavelli: An Environment for Measuring Harmful Agent BehaviorSelected original resource https://arxiv.org/html/2304.03279; sha256:a8cb6e420496e22c40da9d047b91849372fc2105c4077ab0358600b9cacf234f; archived source capture; exact capture time not independently established
Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the Machiavelli Benchmark · §3 Reward Optimization May Produce Machiavellian BehaviorSelected original resource https://arxiv.org/html/2304.03279; sha256:a8cb6e420496e22c40da9d047b91849372fc2105c4077ab0358600b9cacf234f; archived source capture; exact capture time not independently established
Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the Machiavelli Benchmark · §7 DiscussionSelected original resource https://arxiv.org/html/2304.03279; sha256:a8cb6e420496e22c40da9d047b91849372fc2105c4077ab0358600b9cacf234f; archived source capture; exact capture time not independently established

Source checking, extraction, editorial approval and replication are recorded separately. Methodology →

Institutions and roles

Research by University of California, Berkeley Center for AI Safety Carnegie Mellon University Yale University

Authors (10)

Alexander Pan · Jun Shern Chan · Andy Zou · Nathaniel Li · Steven Basart · Thomas Woodside · Jonathan Ng · Hanlin Zhang · Scott Emmons · Dan Hendrycks

Evaluation protocols

MACHIAVELLI →

Cite and reuse

Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.

BibTeX ↓RIS ↓Record + source metadata · JSON ↓Reading coverage · JSON Suggest a correction

Original publication · BibTeX

@misc{machiavelli,
  title = {Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark},
  url = {https://arxiv.org/abs/2304.03279},
  author = {Alexander Pan and Jun Shern Chan and Andy Zou and Nathaniel Li and Steven Basart and Thomas Woodside and Jonathan Ng and Hanlin Zhang and Scott Emmons and Dan Hendrycks},
  year = {2023},
  date = {2023-04-06},
  version = {arXiv:2304.03279v4},
  note = {arXiv:2304.03279v4}
}

NUWA record · BibTeX

@misc{machiavelli-nuwa,
  title = {Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark — NUWA research record},
  url = {https://whitzard.tech/en/nuwa/whitzard-index/work/machiavelli/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--machiavelli.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--machiavelli.json.}
}

Evidence snapshot 2026-10-29.1 · Index updates · RSS

References

[1] Alexander Pan, Jun Shern Chan, et al.. Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark. 2023-04-06. v4 [arXiv]; Author affiliations on the original paper title page; Abstract; §2 Machiavelli: An Environment for Measuring Harmful Agent Behavior