{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"malt","title":"MALT: A Dataset of Natural and Prompted Behaviors That Threaten Eval Integrity","aliases":[],"authors":[],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"dataset","sourceId":"malt-publication","attributionSourceId":"malt-publication","originalUrl":"https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/","publicationDate":"2025-10-14","updatedOn":"2026-10-25","areas":["deception"],"scopeNote":{"en":"Relevance is assigned to the research question and stated conditions of the work. A topic tag is not a risk rating.","zh":"风险关联依据研究问题及明确条件建立，主题标签不代表风险评级。"},"protocolIds":[],"contributions":[{"organizationId":"metr","role":"publisher","sourceId":"malt-publication","locator":"Official publication masthead and article byline"}],"summary":{"en":"Collects manually reviewed agent transcripts and distinguishes naturally observed behavior from prompted examples.","zh":"收集人工审阅的智能体轨迹，区分自然观察行为和提示诱发示例。"},"note":{"question":{"en":"How can monitors for evaluation-integrity failures be checked against labeled agent transcripts?","zh":"如何利用已标注的智能体交互记录检查针对评估完整性失效的监测器？"},"method":{"en":"MALT combines manually reviewed natural examples, prompted or edited sandbagging-like runs, reviewed benign controls and a larger presumed-benign set from HCAST and RE-Bench. Monitor results use the reviewed subset; reasoning-access and post-processed-obfuscation variants are compared.","zh":"MALT 汇集 HCAST 与 RE-Bench 中经人工检查的自然案例、提示或编辑产生的类似沙袋行为运行、已检查的正常对照，以及更大的推定正常集合。监测结果使用已检查子集，并比较推理访问权限和后处理混淆变体。"},"finding":{"en":"Prompted monitors detect many labeled reward-hacking and sandbagging examples. Access to reasoning helps more with the sandbagging labels than with reward hacking; simple post-processed obfuscation has limited effect in the reported tests.","zh":"提示式监测器可以发现许多已标注的奖励黑客与沙袋行为案例。访问推理对沙袋行为标签的帮助大于奖励黑客标签；报告中的简单后处理混淆影响有限。"},"supports":{"en":"The dataset provides a check on monitors used to interpret evaluation runs, while keeping natural behavior, induced behavior and label certainty distinct.","zh":"该数据集可用于检查解释评估运行的监测器，同时区分自然行为、诱导行为与标签确定性。"},"limits":{"en":"Natural concerning examples are few and lack diversity; prompted examples can be conspicuous or unrealistic. Some sandbagging may be invisible in transcripts, and the unreviewed benign set contains label uncertainty. Strong MALT performance is not sufficient or necessary evidence of detecting real-world risks.","zh":"自然出现的可疑案例较少且多样性有限，诱导案例可能明显或不够真实。某些沙袋行为可能无法从记录中识别，未检查的正常集合也存在标签不确定性。MALT 上表现好，并不是发现现实风险的充分或必要证据。"},"connections":null},"resources":[],"noteStatus":{"question":"verified","method":"verified","finding":"verified","supports":"verified","limits":"verified","connections":"limited"},"noteAvailability":{"en":"The substantive sections shown here were checked against the original source. Unfilled sections are omitted; source checking is not experimental replication.","zh":"此处展示的实质性章节已对照原始来源核查；未完成的章节已省略。来源核查不等于实验复现。"},"noteSourceLocator":"Original METR article (14 October 2025), Dataset details; Dataset collection; Monitoring Setup and Results; Limitations","tags":["risk-monitoring"]},"reading":{"id":"malt","title":"MALT: A Dataset of Natural and Prompted Behaviors That Threaten Eval Integrity","en":{"question":"Can manually labeled agent transcripts ground checks for evaluation-integrity failures?","design":"MALT collects 10,919 transcripts from 403 tasks and 21 models, mixing natural and prompted behavior; public and reviewed subsets differ.","findings":["2,690 transcripts were manually reviewed: 1,014 benign and 1,676 with observed behavior.","The remaining 8,229 were not thoroughly checked; monitoring results use the fully reviewed subset."],"interpretation":"NUWA interpretation: a labeled detection dataset should not be mistaken for an unbiased prevalence sample.","limits":"Collection includes anomaly searches and prompted incidents; estimated label error in the unchecked subset was 6%, and public release is task-filtered.","decisionRelevance":"Model recommendation: keep reviewed, public and prompted/unprompted populations distinct in monitoring claims.","takeaway":"NUWA interpretation: a labeled detection dataset should not be mistaken for an unbiased prevalence sample.","connections":[],"scoring":null,"conditions":null,"statistics":"2,690 transcripts were manually reviewed: 1,014 benign and 1,676 with observed behavior. The remaining 8,229 were not thoroughly checked; monitoring results use the fully reviewed subset.","usage":"Model recommendation: keep reviewed, public and prompted/unprompted populations distinct in monitoring claims."},"zh":{"question":"人工标注轨迹能否支撑评测完整性故障检测？","design":"MALT收集21模型、403任务的10,919条轨迹，混合自然与提示行为，公开集和复审集不同。","findings":["人工复审2,690条，其中良性1,014条、有相关行为1,676条。","另8,229条未充分检查，监测结果仅用复审集。"],"interpretation":"NUWA 解读：有标签检测数据不能当无偏发生率样本。","limits":"收集含异常筛查和提示诱发，未检查集估计标签错误6%，公开集也按任务筛选。","decisionRelevance":"模型建议：监测结论中分开复审、公开及提示或自发总体。","takeaway":"NUWA 解读：有标签检测数据不能当无偏发生率样本。","connections":[],"scoring":null,"conditions":null,"statistics":"人工复审2,690条，其中良性1,014条、有相关行为1,676条。 另8,229条未充分检查，监测结果仅用复审集。","usage":"模型建议：监测结论中分开复审、公开及提示或自发总体。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"malt-publication","url":"https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/","locator":"Summary","version":"Selected original resource https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/; sha256:07cbb96eaeb01abc8876dd62deac9039e258b1ddb0d4944e507b830e09148c5f; archived source capture; exact capture time not independently established","artifactSha256":"07cbb96eaeb01abc8876dd62deac9039e258b1ddb0d4944e507b830e09148c5f"},{"sourceId":"malt-publication","url":"https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/","locator":"Dataset details","version":"Selected original resource https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/; sha256:07cbb96eaeb01abc8876dd62deac9039e258b1ddb0d4944e507b830e09148c5f; archived source capture; exact capture time not independently established","artifactSha256":"07cbb96eaeb01abc8876dd62deac9039e258b1ddb0d4944e507b830e09148c5f"},{"sourceId":"malt-publication","url":"https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/","locator":"Dataset collection","version":"Selected original resource https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/; sha256:07cbb96eaeb01abc8876dd62deac9039e258b1ddb0d4944e507b830e09148c5f; archived source capture; exact capture time not independently established","artifactSha256":"07cbb96eaeb01abc8876dd62deac9039e258b1ddb0d4944e507b830e09148c5f"},{"sourceId":"malt-publication","url":"https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/","locator":"Monitoring Setup and Results","version":"Selected original resource https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/; sha256:07cbb96eaeb01abc8876dd62deac9039e258b1ddb0d4944e507b830e09148c5f; archived source capture; exact capture time not independently established","artifactSha256":"07cbb96eaeb01abc8876dd62deac9039e258b1ddb0d4944e507b830e09148c5f"},{"sourceId":"malt-publication","url":"https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/","locator":"Limitations","version":"Selected original resource https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/; sha256:07cbb96eaeb01abc8876dd62deac9039e258b1ddb0d4944e507b830e09148c5f; archived source capture; exact capture time not independently established","artifactSha256":"07cbb96eaeb01abc8876dd62deac9039e258b1ddb0d4944e507b830e09148c5f"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"e73f478bb186f3dc80e5bab1bf1de535c689e1e4a9d424a502c9a31d0d1a1465"}},"sources":[{"id":"malt-publication","title":"MALT: A Dataset of Natural and Prompted Behaviors That Threaten Eval Integrity","url":"https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/","version":"Source capture 2026-10-25","publicationDate":"2025-10-14","datePrecision":"day","type":"research-publication","verifiedOn":"2026-10-25","documentId":"https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors"}]}