NUWA Risk Evaluation · Benchmark
TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents
NUWA research note · independent review pending
NUWA Research Note
Research questionDesign and assumptionsSource findingsNUWA interpretationLimits and contrary evidenceResearch and decision relevanceEvidence updateInstitutions and rolesOriginal resourcesModel-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.
At a glance
Execution-based traces reveal intermediate violations that endpoint scoring misses.
Research question
Can execution traces identify safety violations that final outcomes miss?
Design and assumptions
The benchmark executes 155 sandbox scenarios spanning non-adversarial failures, internal-authority compromise and external attacks. Twelve models are run three times, totaling 5,580 executions. Repeats are averaged within scenario before averaging across eligible scenarios. ASR uses 105 attack-labeled scenarios; the other headline metrics use 155.
Source findings
The mean ASR across the 12 headline models was 0.347. Final-outcome safety can coexist with unsafe intermediate actions. Table 2 preserves TSR, SVR, ASR, STCR and latent violation separately; no event numerators are reconstructed from its rounded ratios.
NUWA interpretation
Task success and safety need joint reporting, with trace evidence adding failure localization. Replaying recorded tool actions diagnoses a recorded trajectory; it is not a counterfactual in which the model makes new decisions.
Limits and contrary evidence
The constructed distribution does not estimate natural incident prevalence. Appendix H bootstrap intervals describe scenario composition; the retained passage does not explicitly state their confidence level.
Research and decision relevance
NUWA recommendation: Use the joint metrics and trace locations to compare intervention points within this source. Do not rank these values against unrelated cyber benchmarks.
Precise source locations
- TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents · §3.6; §5–6; Tables 2,15,17; limitations; text lines 00483Original resource captured 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
- TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents · §3.2 illustrative case-dispute scenario; Table 24Original resource captured 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Source checking, extraction, editorial approval and replication are recorded separately. Methodology →
Evidence update · 2026-10-28
Model-assisted reading · independent evidence and bilingual editorial review pending. Values are source findings or inventory counts, separate from the trend result series.
A source-described sandbox case
In the illustrative case-dispute workflow, an agent completes a useful summary but also accesses protected mock data and sends it to a mock sink after over-trusting an internal message. This is a constructed sandbox example, not a real incident.
The illustrated indicators are TSR=1, SVR=1, ASR=1, latent violation=1 and STCR=0. Task success therefore coexists with both intermediate and endpoint violations.
This explains the metric distinction, not a population rate or causal effect of an intervention. Recorded-action replay localises the boundary crossing without rerunning model decisions.
Original example ↗ · §3.2 illustrative case-dispute scenario; Table 24
Source-bound observations (69)
Counts, rates and contrasts have different units. A sample count is not a number of independent experiments. Unknown numerators and interval levels remain unknown.
| System / metric | Reported value | Population and conditions |
|---|---|---|
| Claude Opus 4.6 TSR | 0.852 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00623,01737Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Opus 4.6 SVR | 0.103 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00623,01737Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Opus 4.6 ASR | 0.095 fraction | Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00623,01737Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Opus 4.6 STCR | 0.755 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00623,01737Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Opus 4.6 Latent | 0.097 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00623,01737Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Sonnet 4.6 TSR | 0.819 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00624,01738,01777,01789Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Sonnet 4.6 SVR | 0.219 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00624,01738,01777,01789Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Sonnet 4.6 ASR | 0.238 fraction | Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00624,01738,01777,01789Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Sonnet 4.6 STCR | 0.639 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00624,01738,01777,01789Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Claude Sonnet 4.6 Latent | 0.194 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00624,01738,01777,01789Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Flash TSR | 0.852 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00625,01739,01773,01785Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Flash SVR | 0.4 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00625,01739,01773,01785Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Flash ASR | 0.533 fraction | Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00625,01739,01773,01785Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Flash STCR | 0.465 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00625,01739,01773,01785Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Flash Latent | 0.387 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00625,01739,01773,01785Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Pro TSR | 0.778 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00626,00862,01554,01740,01774,01786Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Pro SVR | 0.353 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00626,00862,01554,01740,01774,01786Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Pro ASR | 0.467 fraction | Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00626,00862,01554,01740,01774,01786Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Pro STCR | 0.516 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00626,00862,01554,01740,01774,01786Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| DeepSeek V4 Pro Latent | 0.327 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00626,00862,01554,01740,01774,01786Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Flash-Lite Preview TSR | 0.626 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00606,00627,01741Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Flash-Lite Preview SVR | 0.465 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00606,00627,01741Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Flash-Lite Preview ASR | 0.667 fraction | Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00606,00627,01741Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Flash-Lite Preview STCR | 0.31 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00606,00627,01741Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Flash-Lite Preview Latent | 0.432 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00606,00627,01741Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Pro Preview TSR | 0.82 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00628Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Pro Preview SVR | 0.266 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00628Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Pro Preview ASR | 0.402 fraction | Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00628Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Pro Preview STCR | 0.576 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00628Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3.1 Pro Preview Latent | 0.259 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00628Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3 Flash Preview TSR | 0.766 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00629,01742,01778,01790Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3 Flash Preview SVR | 0.383 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00629,01742,01778,01790Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3 Flash Preview ASR | 0.558 fraction | Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00629,01742,01778,01790Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3 Flash Preview STCR | 0.455 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00629,01742,01778,01790Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemini 3 Flash Preview Latent | 0.377 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00629,01742,01778,01790Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemma 4 31B-IT TSR | 0.761 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00630Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemma 4 31B-IT SVR | 0.271 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00630Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemma 4 31B-IT ASR | 0.381 fraction | Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00630Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemma 4 31B-IT STCR | 0.587 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00630Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Gemma 4 31B-IT Latent | 0.258 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00630Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.2 TSR | 0.761 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00631,01743Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.2 SVR | 0.097 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00631,01743Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.2 ASR | 0.143 fraction | Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00631,01743Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.2 STCR | 0.677 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00631,01743Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.2 Latent | 0.097 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00631,01743Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.5 TSR | 0.832 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00605,00632,01744,01775,01787Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.5 SVR | 0.058 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00605,00632,01744,01775,01787Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.5 ASR | 0.076 fraction | Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00605,00632,01744,01775,01787Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.5 STCR | 0.716 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00605,00632,01744,01775,01787Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5.5 Latent | 0.045 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00605,00632,01744,01775,01787Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5 mini TSR | 0.684 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00633,01745,01776,01788Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5 mini SVR | 0.29 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00633,01745,01776,01788Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5 mini ASR | 0.381 fraction | Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00633,01745,01776,01788Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5 mini STCR | 0.503 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00633,01745,01776,01788Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| GPT-5 mini Latent | 0.271 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00633,01745,01776,01788Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| MiniMax M2.5 TSR | 0.858 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00634Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| MiniMax M2.5 SVR | 0.161 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00634Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| MiniMax M2.5 ASR | 0.219 fraction | Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00634Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| MiniMax M2.5 STCR | 0.729 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00634Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| MiniMax M2.5 Latent | 0.142 fraction | Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source{
"condition": "standard prompting",
"temperature": "0.0 where exposed, otherwise provider default",
"tokenCapOverride": null,
"seed": null
}Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00634Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Mean TSR | 0.784 fraction0.742–0.825 · scenario-percentile-bootstrap · level not stated | Conditions, limitations and sourcemodel mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 00602,01678Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Mean SVR | 0.256 fraction0.213–0.3 · scenario-percentile-bootstrap · level not stated | Conditions, limitations and sourcemodel mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 01679Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Mean ASR | 0.347 fraction0.294–0.4 · scenario-percentile-bootstrap · level not stated | Conditions, limitations and sourcemodel mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 00153,00603,01680Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Mean STCR | 0.577 fraction0.527–0.627 · scenario-percentile-bootstrap · level not stated | Conditions, limitations and sourcemodel mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 01681Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Mean latent violation rate | 0.24 fraction0.199–0.284 · scenario-percentile-bootstrap · level not stated | Conditions, limitations and sourcemodel mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 00859,01682Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Internal-authority SVR | 0.424 fraction0.347–0.499 · scenario-percentile-bootstrap · level not stated | Conditions, limitations and sourcemodel mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 01683Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| External-attack SVR | 0.242 fraction0.184–0.305 · scenario-percentile-bootstrap · level not stated | Conditions, limitations and sourcemodel mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 01684Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Internal minus external SVR | 0.182 fraction-difference0.083–0.278 · scenario-percentile-bootstrap · level not stated | Conditions, limitations and sourcemodel mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 01685Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
| Prompt reminder∆ASR | -0.107 fraction-difference-0.132–-0.083 · scenario-percentile-bootstrap · level not stated | Conditions, limitations and sourcemodel mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 01689Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 |
Related reading
NUWA reading connections; these do not imply citation, replication or score equivalence.
Institutions and roles
Authors (5)
Yuxiang He · Jian Zhao · Yufei Liu · Yuchen Yuan · Xuelong Li
Evaluation protocols
Cite and reuse
Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.
Original publication · BibTeX
@misc{traceprobe,
title = {TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents},
url = {https://aclanthology.org/2026.emnlp-main.1233.pdf},
author = {Yuxiang He and Jian Zhao and Yufei Liu and Yuchen Yuan and Xuelong Li},
version = {Original resource captured 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828},
note = {Original resource captured 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828}
}
NUWA record · BibTeX
@misc{traceprobe-nuwa,
title = {TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents — NUWA research record},
url = {https://whitzard.tech/en/nuwa/whitzard-index/work/traceprobe/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--traceprobe.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--traceprobe.json.}
}
Evidence snapshot 2026-10-29.1 · Index updates · RSS
References
- [1] TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents ; §3.6; §5–6; Tables 2,15,17; limitations; text lines 00483; §3.2 illustrative case-dispute scenario; Table 24