{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"misuse-safety-case","title":"An example safety case for safeguards against misuse","aliases":[],"authors":[],"primaryFunction":"governance","secondaryFunctions":[],"kind":"technical-report","sourceId":"misuse-safety-case-publication","attributionSourceId":"misuse-safety-case-publication","originalUrl":"https://www.aisi.gov.uk/research/an-example-safety-case-for-safeguards-against-misuse","publicationDate":"2025-06-05","updatedOn":"2026-10-25","areas":[],"scopeNote":{"en":"Relevance is assigned to the research question and stated conditions of the work. A topic tag is not a risk rating.","zh":"风险关联依据研究问题及明确条件建立，主题标签不代表风险评级。"},"protocolIds":[],"contributions":[{"organizationId":"aisi","role":"publisher","sourceId":"misuse-safety-case-publication","locator":"Official publication masthead and article byline"}],"summary":{"en":"Constructs an illustrative argument linking safeguard red-teaming evidence, attacker effort and deployment-risk decisions.","zh":"构造示例安全论证，将防护红队证据、攻击者投入与部署风险决策联系起来。"},"note":{"question":null,"method":null,"finding":null,"supports":null,"limits":null,"connections":null},"resources":[{"label":"Read the full paper","url":"https://arxiv.org/abs/2505.18003"}],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"A substantive NUWA research note is not yet available. The repeated draft sections were removed pending review of the original methods, findings and limitations; use the original publication for the research account.","zh":"实质性的 NUWA 研究笔记尚未完成。重复草稿章节已移除，待核查原始方法、发现与限制；研究内容请以原始出版物为准。"},"noteSourceLocator":"No complete section-level method/results/limitations review completed for this note; cached original page and any chart audit retained as discovery evidence.","tags":[]},"reading":{"id":"misuse-safety-case","title":"An example safety case for safeguards against misuse","en":{"takeaway":"This paper is a worked safety argument for a hypothetical assistant, not a certification of a deployed system. It connects safeguard testing to a risk model and explicit operational commitments.","question":"How could evaluations of misuse safeguards support a defensible claim that deployment adds less than a specified level of harm?","design":"The proposed case defines a harm pathway and threshold, constructs representative requests, and red-teams safeguards to estimate effort-versus-success curves. A quantitative uplift model links those curves to assumptions about misuse actors, background risk and non-AI barriers. Continuing evaluation and timely response are part of the argument, not optional additions to a one-time score.","scoring":"There is no benchmark leaderboard endpoint for a real deployment. The argument estimates incremental post-mitigation risk relative to a no-deployment baseline, under explicit pathway and actor assumptions. Red-team effort is an intermediate measurement, not the final probability of harm.","findings":["The paper decomposes the top claim into conditions on pathway coverage, conservative evaluations, risk forecasting and the developer’s ability to respond (§2.7).","Separate assessment of safeguards and aggregation require justification; tests must conservatively represent deployment rather than merely accumulate jailbreak examples (§2.4).","The discussion identifies difficult cases, including pathways where few requests could enable substantial harm; broad transfer to such cases is not assumed (§3)."],"interpretation":"The contribution is an explicit chain from observations to a decision, exposing assumptions that would otherwise stay implicit. The illustrative numerical model does not validate its inputs or prove that a particular assistant meets a threshold.","limits":"Threat coverage, request representativeness, actor effort and willingness, and response delay are uncertain. Red-team capabilities and incentives must upper-bound the relevant misuse population in the intended sense. A lower-risk proxy needs justification rather than automatic substitution. The paper’s hypothetical procedural commitments are not evidence that any developer follows them.","connections":["The source builds on safety-case methods and AISI safeguard evaluation principles. AI Control addresses intentional model subversion, a different threat from an external actor evading misuse safeguards; one argument cannot silently cover the other."],"decisionRelevance":"Model recommendation: write each inferential step and response commitment as an independently checkable claim; keep illustrative model assumptions separate from observed evaluation facts.","statistics":"The paper proposes effort curves, extrapolation and an uplift model. It does not report a validated deployment harm-rate estimate for an actual assistant."},"zh":{"takeaway":"本文为假设性助手构建安全论证，并非给已部署系统认证；它把防护评测、风险模型与明确的运行承诺连接起来。","question":"如何用滥用防护评测支持“部署新增伤害低于特定阈值”的可辩护主张？","design":"方案先定义伤害路径与阈值、构造有代表性的请求，再用红队估计投入与成功的关系。量化增益模型将这些曲线连接到滥用者、背景风险和非 AI 障碍的假设。持续评测与及时响应是论证组成部分，不能简化为一次评分。","scoring":"本文没有真实部署的基准排行榜终点；它在明确路径和行为者假设下，比较防护后与未部署基线的增量风险。红队投入是中间测量，并非最终伤害概率。","findings":["§2.7 把顶层主张分解为路径覆盖、保守评测、风险预测和开发者响应能力等条件。","分别评估防护及汇总需要依据，测试应保守代表部署，不能只是累积越狱示例（§2.4）。","讨论明确指出少量请求即可造成重大伤害等困难情形，不预设论证可以普遍迁移（§3）。"],"interpretation":"贡献在于显式写出从观测到决策的推理链，让隐含假设可见；示例量化模型并未验证输入，也未证明某个助手达到阈值。","limits":"威胁覆盖、请求代表性、行为者投入与意愿、响应延迟均有不确定性。红队能力和激励需在对应意义上保守代表滥用群体，低风险代理任务也必须论证其适用性。假设性的程序承诺不是开发者实际遵守的证据。","connections":["来源建立在安全案例方法及 AISI 防护评测原则之上。AI Control 研究模型蓄意破坏，与外部行为者绕过防护不同，一种论证不能悄悄替代另一种。"],"decisionRelevance":"模型建议：把每步推理和响应承诺写成可独立核查的主张，区分示例模型假设与实际评测事实。","statistics":"论文提出投入曲线、外推与增益模型，没有报告实际助手经验证的部署伤害率。"},"figureRefs":[],"assertionPolicy":{"design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","connections":"nuwa-interpretation-with-source-refs","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"misuse-safety-case-publication","url":"https://arxiv.org/html/2505.18003","locator":"§2.1 Safety case objectives and definitions","version":"Linked original captured 2026-10-27T09:55:34.683086+00:00; sha256:780bb7c2e2495d6b91fa561fd3fc05045444dde54f8b71815212825b9fe53079","artifactSha256":"780bb7c2e2495d6b91fa561fd3fc05045444dde54f8b71815212825b9fe53079"},{"sourceId":"misuse-safety-case-publication","url":"https://arxiv.org/html/2505.18003","locator":"§2.2 Threat modelling; §2.3 Misuse safeguards","version":"Linked original captured 2026-10-27T09:55:34.683086+00:00; sha256:780bb7c2e2495d6b91fa561fd3fc05045444dde54f8b71815212825b9fe53079","artifactSha256":"780bb7c2e2495d6b91fa561fd3fc05045444dde54f8b71815212825b9fe53079"},{"sourceId":"misuse-safety-case-publication","url":"https://arxiv.org/html/2505.18003","locator":"§2.4 The safeguard evaluation methodology","version":"Linked original captured 2026-10-27T09:55:34.683086+00:00; sha256:780bb7c2e2495d6b91fa561fd3fc05045444dde54f8b71815212825b9fe53079","artifactSha256":"780bb7c2e2495d6b91fa561fd3fc05045444dde54f8b71815212825b9fe53079"},{"sourceId":"misuse-safety-case-publication","url":"https://arxiv.org/html/2505.18003","locator":"§2.5 The uplift model","version":"Linked original captured 2026-10-27T09:55:34.683086+00:00; sha256:780bb7c2e2495d6b91fa561fd3fc05045444dde54f8b71815212825b9fe53079","artifactSha256":"780bb7c2e2495d6b91fa561fd3fc05045444dde54f8b71815212825b9fe53079"},{"sourceId":"misuse-safety-case-publication","url":"https://arxiv.org/html/2505.18003","locator":"§2.6 Responding to evaluation results; §2.7 Safety case","version":"Linked original captured 2026-10-27T09:55:34.683086+00:00; sha256:780bb7c2e2495d6b91fa561fd3fc05045444dde54f8b71815212825b9fe53079","artifactSha256":"780bb7c2e2495d6b91fa561fd3fc05045444dde54f8b71815212825b9fe53079"},{"sourceId":"misuse-safety-case-publication","url":"https://arxiv.org/html/2505.18003","locator":"§3 Discussion; §4 Conclusion","version":"Linked original captured 2026-10-27T09:55:34.683086+00:00; sha256:780bb7c2e2495d6b91fa561fd3fc05045444dde54f8b71815212825b9fe53079","artifactSha256":"780bb7c2e2495d6b91fa561fd3fc05045444dde54f8b71815212825b9fe53079"},{"sourceId":"misuse-safety-case-publication","url":"https://arxiv.org/html/2505.18003","locator":"Appendix D The red team; Appendix F The dataset of harmful requests","version":"Linked original captured 2026-10-27T09:55:34.683086+00:00; sha256:780bb7c2e2495d6b91fa561fd3fc05045444dde54f8b71815212825b9fe53079","artifactSha256":"780bb7c2e2495d6b91fa561fd3fc05045444dde54f8b71815212825b9fe53079"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":"full-text","readingCoverage":"selected-methods-results-limitations-sections","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"b07820288c83259f960d76a60018d5401f75e230ddd2e66baf79a3f61f73dda0"}},"sources":[{"id":"misuse-safety-case-publication","title":"An example safety case for safeguards against misuse","url":"https://www.aisi.gov.uk/research/an-example-safety-case-for-safeguards-against-misuse","version":"Source capture 2026-10-25","publicationDate":"2025-06-05","datePrecision":"day","type":"research-publication","verifiedOn":"2026-10-25","documentId":"https://www.aisi.gov.uk/research/an-example-safety-case-for-safeguards-against-misuse"}]}