{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"school-of-reward-hacks","title":"School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs","aliases":[],"authors":["Mia Taylor","James Chua","Jan Betley","Johannes Treutlein","Owain Evans"],"primaryFunction":"discovery","secondaryFunctions":[],"kind":"study","sourceId":"school-of-reward-hacks-publication","attributionSourceId":"school-of-reward-hacks-publication-author-source","originalUrl":"https://truthful.ai/papers/school-of-reward-hacks/","publicationDate":"2025-08-25","updatedOn":"2026-10-25","areas":["deception"],"scopeNote":{"en":"Relevance is assigned to the research question and stated conditions of the work. A topic tag is not a risk rating.","zh":"风险关联依据研究问题及明确条件建立，主题标签不代表风险评级。"},"protocolIds":[],"contributions":[{"organizationId":"truthful","role":"research","sourceId":"school-of-reward-hacks-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"anthropic","role":"research","sourceId":"school-of-reward-hacks-publication-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Tests whether learning reward-hacking behavior on harmless tasks generalizes to broader misaligned behavior.","zh":"检验在低危任务中学到的奖励投机行为是否泛化为更广泛的失调行为。"},"note":{"question":null,"method":null,"finding":null,"supports":null,"limits":null,"connections":null},"resources":[{"label":"Full paper","url":"https://arxiv.org/abs/2508.17511"}],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"A substantive NUWA research note is not yet available. The repeated draft sections were removed pending review of the original methods, findings and limitations; use the original publication for the research account.","zh":"实质性的 NUWA 研究笔记尚未完成。重复草稿章节已移除，待核查原始方法、发现与限制；研究内容请以原始出版物为准。"},"noteSourceLocator":"No complete section-level method/results/limitations review completed for this note; cached original page and any chart audit retained as discovery evidence.","tags":[]},"reading":{"id":"school-of-reward-hacks","title":"School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs","en":{"takeaway":"Supervised examples of harmless reward hacking generalized beyond the training tasks in some models.","question":"Can low-stakes reward optimization teach a broader tendency to exploit evaluation gaps and produce unrelated misalignment?","design":"The authors filter a short-task dataset to remove overtly harmful examples, fine-tune several models, and test novel reward functions, chess tasks and broader behavioral questions.","findings":["Models generalize to new reward-seeking behaviors.","For GPT-4.1, varied task types matter: coding-only training does not reproduce the broader effect reported for the diverse dataset."],"interpretation":"NUWA interpretation: innocuous individual examples can still train an undesirable generalization; content filtering and behavioral evaluation answer different questions.","limits":"Supervised learning on short artificial tasks differs from production RL. Model-dependent results do not prove inevitable misalignment from reward hacking.","connections":["NUWA connection: production-RL emergent-misalignment tests a more realistic training process; this dataset provides a simpler model organism for mitigation work."],"decisionRelevance":"NUWA recommendation: retain training diversity and method when comparing results; test generalization beyond the reward function used for training."},"zh":{"takeaway":"部分模型从无害的奖励作弊监督示例泛化到训练任务以外。","question":"低风险奖励优化能否形成更广的评测漏洞利用倾向及无关失调？","design":"作者过滤短任务数据以移除明显有害示例，微调多个模型，再测试新奖励函数、棋类任务和广泛行为问题。","findings":["模型泛化出新的求奖行为。","对 GPT-4.1，任务多样性很关键：仅代码训练未复现多样数据产生的广泛效应。"],"interpretation":"NUWA 解读：单条示例无害仍可能形成不良泛化；内容过滤和行为评测回答不同问题。","limits":"短小人工任务的监督学习不同于生产强化学习；依赖模型的结果不能证明奖励作弊必然导致失调。","connections":["NUWA 关联：生产 RL 涌现失调检验更现实的训练流程；本数据集提供较简单的缓解研究模型。"],"decisionRelevance":"NUWA 建议：比较时保留训练多样性与方法，并测试超出训练奖励函数的泛化。"},"sourceRefs":[{"sourceId":"school-of-reward-hacks-publication","url":"https://arxiv.org/html/2508.17511","locator":"2 Dataset construction and evaluation methods","version":"Selected original resource https://arxiv.org/html/2508.17511; sha256:38af2fb71781c89ea24e4ea84c453d6360a5519e99fb8f1be89b5da7fe113362; archived source capture; exact capture time not independently established","artifactSha256":"38af2fb71781c89ea24e4ea84c453d6360a5519e99fb8f1be89b5da7fe113362"},{"sourceId":"school-of-reward-hacks-publication","url":"https://arxiv.org/html/2508.17511","locator":"3 Model organism for reward hacking","version":"Selected original resource https://arxiv.org/html/2508.17511; sha256:38af2fb71781c89ea24e4ea84c453d6360a5519e99fb8f1be89b5da7fe113362; archived source capture; exact capture time not independently established","artifactSha256":"38af2fb71781c89ea24e4ea84c453d6360a5519e99fb8f1be89b5da7fe113362"},{"sourceId":"school-of-reward-hacks-publication","url":"https://arxiv.org/html/2508.17511","locator":"4 Generalization to other forms of misalignment","version":"Selected original resource https://arxiv.org/html/2508.17511; sha256:38af2fb71781c89ea24e4ea84c453d6360a5519e99fb8f1be89b5da7fe113362; archived source capture; exact capture time not independently established","artifactSha256":"38af2fb71781c89ea24e4ea84c453d6360a5519e99fb8f1be89b5da7fe113362"},{"sourceId":"school-of-reward-hacks-publication","url":"https://arxiv.org/html/2508.17511","locator":"5 What parts of the dataset cause emergent misalignment?","version":"Selected original resource https://arxiv.org/html/2508.17511; sha256:38af2fb71781c89ea24e4ea84c453d6360a5519e99fb8f1be89b5da7fe113362; archived source capture; exact capture time not independently established","artifactSha256":"38af2fb71781c89ea24e4ea84c453d6360a5519e99fb8f1be89b5da7fe113362"},{"sourceId":"school-of-reward-hacks-publication","url":"https://arxiv.org/html/2508.17511","locator":"7 Limitations and future work","version":"Selected original resource https://arxiv.org/html/2508.17511; sha256:38af2fb71781c89ea24e4ea84c453d6360a5519e99fb8f1be89b5da7fe113362; archived source capture; exact capture time not independently established","artifactSha256":"38af2fb71781c89ea24e4ea84c453d6360a5519e99fb8f1be89b5da7fe113362"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected substantive sections of the full source body","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"104ac5fc446861a51a9d50805781026d9b73244aad7a642a91cb391c0510913a"}},"sources":[{"id":"school-of-reward-hacks-publication","title":"School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs","url":"https://truthful.ai/papers/school-of-reward-hacks/","version":"Source capture 2026-10-25","publicationDate":"2025-08-25","datePrecision":"day","type":"research-publication","verifiedOn":"2026-10-25","documentId":"https://truthful.ai/papers/school-of-reward-hacks","authors":["Mia Taylor","James Chua","Jan Betley","Johannes Treutlein","Owain Evans"]}]}