NUWA 风险评测 · 评测基准
TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents
NUWA 研究解读 · 独立复核待完成
NUWA Research Note
研究问题研究设计与假设来源发现NUWA 解读局限与相反证据研究与决策意义证据补充机构与角色原始资源模型辅助整理 · 选读原始全文或官方文章中的相关章节；尚未完成逐章节审计。
研究速读
155 场景、12 模型、每设置 3 次运行；分别报告任务成功、终态违规、攻击子集成功和潜在中间违规。
研究问题
终态安全是否掩盖中途违规？
研究设计与假设
155 个可执行沙箱场景分为非对抗失效、内部权限受损和外部攻击。12 个模型各运行三次，共 5,580 次执行。指标先在场景内平均重复，再跨适用场景平均；ASR 只使用 105 个攻击标注场景，其余主指标使用 155 个场景。
来源发现
12 个主模型的平均 ASR 为 0.347。终点安全的运行仍可能包含不安全中间动作。表 2 分别保留 TSR、SVR、ASR、STCR 与潜在违规，不从舍入比例反推事件分子。
NUWA 解读
终点成功与安全必须联合报告，轨迹可补充失效位置。工具动作回放能够定位已有轨迹中的触发点，但不是让模型重新决策的因果反事实。
局限与相反证据
ASR 仅针对攻击标记子集；回放重执行既定工具调用，不重新运行模型，不能证明反事实因果。
研究与决策意义
NUWA 建议：新增过程证据层；保留主试验、配对提醒、压力测试和回放各自分母。
精确来源位置
- TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents · §3.6; §5–6; Tables 2,15,17; limitations; text lines 00483Original resource captured 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
- TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents · §3.2 illustrative case-dispute scenario; Table 24Original resource captured 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
来源核查、数据提取、编辑批准与实验复现分别记录。 方法说明 →
证据补充 · 2026-10-28
模型辅助整理 · 独立证据及双语编辑复核待完成。以下数值为来源发现或材料计数，与趋势结果序列分开保存。
一项来源描述的沙箱案例
论文的争议处理示例中，智能体完成有用摘要，却在过度信任内部消息后读取受保护的模拟数据并发往模拟接收端。这是构造沙箱案例，不是现实事故。
示例指标为 TSR=1、SVR=1、ASR=1、潜在违规=1、STCR=0；任务成功与中间、终点违规并存。
案例解释指标之间的区别，不代表总体发生率或干预因果效果。既定动作回放能够定位越界位置，不重新运行模型决策。
原始案例 ↗ · §3.2 illustrative case-dispute scenario; Table 24
原文数值与条件（69 条）
计数、比例与对比效应使用不同单位。样本数量不等于独立实验数量；未披露的分子和区间水平保持未知。
| 系统／指标 | 原文数值 | 总体与条件 |
|---|---|---|
| Claude Opus 4.6 TSR | 0.852 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00623,01737Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Opus 4.6 SVR | 0.103 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00623,01737Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Opus 4.6 ASR | 0.095 比例 | 分母: 105 攻击标注场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 315 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00623,01737Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Opus 4.6 STCR | 0.755 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00623,01737Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Opus 4.6 Latent | 0.097 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00623,01737Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Sonnet 4.6 TSR | 0.819 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00624,01738,01777,01789Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Sonnet 4.6 SVR | 0.219 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00624,01738,01777,01789Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Sonnet 4.6 ASR | 0.238 比例 | 分母: 105 攻击标注场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 315 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00624,01738,01777,01789Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Sonnet 4.6 STCR | 0.639 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00624,01738,01777,01789Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Sonnet 4.6 Latent | 0.194 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00624,01738,01777,01789Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Flash TSR | 0.852 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00625,01739,01773,01785Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Flash SVR | 0.4 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00625,01739,01773,01785Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Flash ASR | 0.533 比例 | 分母: 105 攻击标注场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 315 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00625,01739,01773,01785Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Flash STCR | 0.465 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00625,01739,01773,01785Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Flash Latent | 0.387 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00625,01739,01773,01785Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Pro TSR | 0.778 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00626,00862,01554,01740,01774,01786Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Pro SVR | 0.353 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00626,00862,01554,01740,01774,01786Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Pro ASR | 0.467 比例 | 分母: 105 攻击标注场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 315 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00626,00862,01554,01740,01774,01786Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Pro STCR | 0.516 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00626,00862,01554,01740,01774,01786Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Pro Latent | 0.327 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00626,00862,01554,01740,01774,01786Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Flash-Lite Preview TSR | 0.626 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00606,00627,01741Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Flash-Lite Preview SVR | 0.465 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00606,00627,01741Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Flash-Lite Preview ASR | 0.667 比例 | 分母: 105 攻击标注场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 315 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00606,00627,01741Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Flash-Lite Preview STCR | 0.31 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00606,00627,01741Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Flash-Lite Preview Latent | 0.432 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00606,00627,01741Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Pro Preview TSR | 0.82 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00628Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Pro Preview SVR | 0.266 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00628Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Pro Preview ASR | 0.402 比例 | 分母: 105 攻击标注场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 315 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00628Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Pro Preview STCR | 0.576 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00628Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Pro Preview Latent | 0.259 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00628Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3 Flash Preview TSR | 0.766 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00629,01742,01778,01790Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3 Flash Preview SVR | 0.383 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00629,01742,01778,01790Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3 Flash Preview ASR | 0.558 比例 | 分母: 105 攻击标注场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 315 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00629,01742,01778,01790Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3 Flash Preview STCR | 0.455 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00629,01742,01778,01790Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3 Flash Preview Latent | 0.377 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00629,01742,01778,01790Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemma 4 31B-IT TSR | 0.761 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00630Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemma 4 31B-IT SVR | 0.271 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00630Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemma 4 31B-IT ASR | 0.381 比例 | 分母: 105 攻击标注场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 315 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00630Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemma 4 31B-IT STCR | 0.587 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00630Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemma 4 31B-IT Latent | 0.258 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00630Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.2 TSR | 0.761 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00631,01743Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.2 SVR | 0.097 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00631,01743Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.2 ASR | 0.143 比例 | 分母: 105 攻击标注场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 315 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00631,01743Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.2 STCR | 0.677 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00631,01743Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.2 Latent | 0.097 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00631,01743Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.5 TSR | 0.832 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00605,00632,01744,01775,01787Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.5 SVR | 0.058 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00605,00632,01744,01775,01787Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.5 ASR | 0.076 比例 | 分母: 105 攻击标注场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 315 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00605,00632,01744,01775,01787Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.5 STCR | 0.716 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00605,00632,01744,01775,01787Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.5 Latent | 0.045 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00605,00632,01744,01775,01787Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5 mini TSR | 0.684 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00633,01745,01776,01788Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5 mini SVR | 0.29 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00633,01745,01776,01788Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5 mini ASR | 0.381 比例 | 分母: 105 攻击标注场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 315 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00633,01745,01776,01788Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5 mini STCR | 0.503 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00633,01745,01776,01788Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5 mini Latent | 0.271 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00633,01745,01776,01788Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| MiniMax M2.5 TSR | 0.858 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00634Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| MiniMax M2.5 SVR | 0.161 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00634Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| MiniMax M2.5 ASR | 0.219 比例 | 分母: 105 攻击标注场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 315 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00634Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| MiniMax M2.5 STCR | 0.729 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00634Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| MiniMax M2.5 Latent | 0.142 比例 | 分母: 155 场景 · scenario mean after averaging 3 repeats 每场景重复: 3 · 执行: 465 (NUWA 计算) 条件、限制与来源{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. 原文 ↗ · Table2 PDFp7; §3.6; Table9; text lines 00634Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Mean TSR | 0.784 比例0.742–0.825 · scenario-percentile-bootstrap · 水平未说明 | 条件、限制与来源model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. 原文 ↗ · Table17 PDFp18; Appendix H; text lines 00602,01678Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Mean SVR | 0.256 比例0.213–0.3 · scenario-percentile-bootstrap · 水平未说明 | 条件、限制与来源model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. 原文 ↗ · Table17 PDFp18; Appendix H; text lines 01679Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Mean ASR | 0.347 比例0.294–0.4 · scenario-percentile-bootstrap · 水平未说明 | 条件、限制与来源model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. 原文 ↗ · Table17 PDFp18; Appendix H; text lines 00153,00603,01680Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Mean STCR | 0.577 比例0.527–0.627 · scenario-percentile-bootstrap · 水平未说明 | 条件、限制与来源model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. 原文 ↗ · Table17 PDFp18; Appendix H; text lines 01681Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Mean latent violation rate | 0.24 比例0.199–0.284 · scenario-percentile-bootstrap · 水平未说明 | 条件、限制与来源model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. 原文 ↗ · Table17 PDFp18; Appendix H; text lines 00859,01682Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Internal-authority SVR | 0.424 比例0.347–0.499 · scenario-percentile-bootstrap · 水平未说明 | 条件、限制与来源model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. 原文 ↗ · Table17 PDFp18; Appendix H; text lines 01683Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| External-attack SVR | 0.242 比例0.184–0.305 · scenario-percentile-bootstrap · 水平未说明 | 条件、限制与来源model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. 原文 ↗ · Table17 PDFp18; Appendix H; text lines 01684Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Internal minus external SVR | 0.182 比例差0.083–0.278 · scenario-percentile-bootstrap · 水平未说明 | 条件、限制与来源model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. 原文 ↗ · Table17 PDFp18; Appendix H; text lines 01685Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Prompt reminder∆ASR | -0.107 比例差-0.132–-0.083 · scenario-percentile-bootstrap · 水平未说明 | 条件、限制与来源model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. 原文 ↗ · Table17 PDFp18; Appendix H; text lines 01689Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
关联阅读
NUWA 阅读关联；不据此声称原文引用、实验复现或成绩等价。
机构与角色
作者 (5)
Yuxiang He · Jian Zhao · Yufei Liu · Yuchen Yuan · Xuelong Li
评测协议
引用与复用
引用研究发现时请引用原始出版物；引用 NUWA 的整理与解释时，单独引用本站记录及对应快照。
原始出版物 · BibTeX
@misc{traceprobe,
title = {TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents},
url = {https://aclanthology.org/2026.emnlp-main.1233.pdf},
author = {Yuxiang He and Jian Zhao and Yufei Liu and Yuchen Yuan and Xuelong Li},
version = {Original resource captured 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828},
note = {Original resource captured 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828}
}
NUWA 记录 · BibTeX
@misc{traceprobe-nuwa,
title = {TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents — NUWA research record},
url = {https://whitzard.tech/nuwa/whitzard-index/work/traceprobe/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--traceprobe.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--traceprobe.json.}
}
证据快照 2026-10-29.1 · 索引更新 · RSS
参考文献
- [1] TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents ; §3.6; §5–6; Tables 2,15,17; limitations; text lines 00483; §3.2 illustrative case-dispute scenario; Table 24