Evaluation dossier

TraceProbe

Execution-based traces reveal intermediate violations that endpoint scoring misses.

Affiliation not indexed in this review / Original resource captured 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828 / Behavior

Sources and provenance ↓ Read the research narrative →

Evidence update · 2026-10-28

Model-assisted reading · independent evidence and bilingual editorial review pending. Values are source findings or inventory counts, separate from the trend result series.

A source-described sandbox case

In the illustrative case-dispute workflow, an agent completes a useful summary but also accesses protected mock data and sends it to a mock sink after over-trusting an internal message. This is a constructed sandbox example, not a real incident.

The illustrated indicators are TSR=1, SVR=1, ASR=1, latent violation=1 and STCR=0. Task success therefore coexists with both intermediate and endpoint violations.

This explains the metric distinction, not a population rate or causal effect of an intervention. Recorded-action replay localises the boundary crossing without rerunning model decisions.

Original example ↗ · §3.2 illustrative case-dispute scenario; Table 24

Source-bound observations (69)

Counts, rates and contrasts have different units. A sample count is not a number of independent experiments. Unknown numerators and interval levels remain unknown.

System / metric	Reported value	Population and conditions
Claude Opus 4.6 TSR	0.852 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00623,01737Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Claude Opus 4.6 SVR	0.103 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00623,01737Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Claude Opus 4.6 ASR	0.095 fraction	Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00623,01737Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Claude Opus 4.6 STCR	0.755 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00623,01737Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Claude Opus 4.6 Latent	0.097 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00623,01737Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Claude Sonnet 4.6 TSR	0.819 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00624,01738,01777,01789Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Claude Sonnet 4.6 SVR	0.219 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00624,01738,01777,01789Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Claude Sonnet 4.6 ASR	0.238 fraction	Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00624,01738,01777,01789Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Claude Sonnet 4.6 STCR	0.639 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00624,01738,01777,01789Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Claude Sonnet 4.6 Latent	0.194 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00624,01738,01777,01789Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
DeepSeek V4 Flash TSR	0.852 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00625,01739,01773,01785Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
DeepSeek V4 Flash SVR	0.4 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00625,01739,01773,01785Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
DeepSeek V4 Flash ASR	0.533 fraction	Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00625,01739,01773,01785Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
DeepSeek V4 Flash STCR	0.465 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00625,01739,01773,01785Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
DeepSeek V4 Flash Latent	0.387 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00625,01739,01773,01785Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
DeepSeek V4 Pro TSR	0.778 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00626,00862,01554,01740,01774,01786Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
DeepSeek V4 Pro SVR	0.353 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00626,00862,01554,01740,01774,01786Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
DeepSeek V4 Pro ASR	0.467 fraction	Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00626,00862,01554,01740,01774,01786Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
DeepSeek V4 Pro STCR	0.516 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00626,00862,01554,01740,01774,01786Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
DeepSeek V4 Pro Latent	0.327 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00626,00862,01554,01740,01774,01786Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemini 3.1 Flash-Lite Preview TSR	0.626 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00606,00627,01741Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemini 3.1 Flash-Lite Preview SVR	0.465 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00606,00627,01741Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemini 3.1 Flash-Lite Preview ASR	0.667 fraction	Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00606,00627,01741Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemini 3.1 Flash-Lite Preview STCR	0.31 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00606,00627,01741Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemini 3.1 Flash-Lite Preview Latent	0.432 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00606,00627,01741Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemini 3.1 Pro Preview TSR	0.82 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00628Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemini 3.1 Pro Preview SVR	0.266 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00628Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemini 3.1 Pro Preview ASR	0.402 fraction	Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00628Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemini 3.1 Pro Preview STCR	0.576 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00628Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemini 3.1 Pro Preview Latent	0.259 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00628Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemini 3 Flash Preview TSR	0.766 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00629,01742,01778,01790Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemini 3 Flash Preview SVR	0.383 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00629,01742,01778,01790Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemini 3 Flash Preview ASR	0.558 fraction	Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00629,01742,01778,01790Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemini 3 Flash Preview STCR	0.455 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00629,01742,01778,01790Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemini 3 Flash Preview Latent	0.377 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00629,01742,01778,01790Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemma 4 31B-IT TSR	0.761 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00630Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemma 4 31B-IT SVR	0.271 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00630Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemma 4 31B-IT ASR	0.381 fraction	Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00630Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemma 4 31B-IT STCR	0.587 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00630Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Gemma 4 31B-IT Latent	0.258 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00630Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
GPT-5.2 TSR	0.761 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00631,01743Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
GPT-5.2 SVR	0.097 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00631,01743Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
GPT-5.2 ASR	0.143 fraction	Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00631,01743Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
GPT-5.2 STCR	0.677 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00631,01743Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
GPT-5.2 Latent	0.097 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00631,01743Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
GPT-5.5 TSR	0.832 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00605,00632,01744,01775,01787Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
GPT-5.5 SVR	0.058 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00605,00632,01744,01775,01787Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
GPT-5.5 ASR	0.076 fraction	Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00605,00632,01744,01775,01787Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
GPT-5.5 STCR	0.716 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00605,00632,01744,01775,01787Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
GPT-5.5 Latent	0.045 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00605,00632,01744,01775,01787Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
GPT-5 mini TSR	0.684 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00633,01745,01776,01788Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
GPT-5 mini SVR	0.29 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00633,01745,01776,01788Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
GPT-5 mini ASR	0.381 fraction	Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00633,01745,01776,01788Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
GPT-5 mini STCR	0.503 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00633,01745,01776,01788Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
GPT-5 mini Latent	0.271 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00633,01745,01776,01788Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
MiniMax M2.5 TSR	0.858 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00634Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
MiniMax M2.5 SVR	0.161 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00634Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
MiniMax M2.5 ASR	0.219 fraction	Denominator: 105 attack-labeled-scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 315 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00634Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
MiniMax M2.5 STCR	0.729 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00634Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
MiniMax M2.5 Latent	0.142 fraction	Denominator: 155 scenarios · scenario mean after averaging 3 repeats Repeats per scenario: 3 · executions: 465 (NUWA arithmetic) Conditions, limitations and source { "condition": "standard prompting", "temperature": "0.0 where exposed, otherwise provider default", "tokenCapOverride": null, "seed": null } Constructed scenario distribution, not natural prevalence; no raw numerator inferred from rounded rate. Original source ↗ · Table2 PDFp7; §3.6; Table9; text lines 00634Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Mean TSR	0.784 fraction0.742–0.825 · scenario-percentile-bootstrap · level not stated	Conditions, limitations and source model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 00602,01678Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Mean SVR	0.256 fraction0.213–0.3 · scenario-percentile-bootstrap · level not stated	Conditions, limitations and source model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 01679Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Mean ASR	0.347 fraction0.294–0.4 · scenario-percentile-bootstrap · level not stated	Conditions, limitations and source model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 00153,00603,01680Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Mean STCR	0.577 fraction0.527–0.627 · scenario-percentile-bootstrap · level not stated	Conditions, limitations and source model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 01681Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Mean latent violation rate	0.24 fraction0.199–0.284 · scenario-percentile-bootstrap · level not stated	Conditions, limitations and source model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 00859,01682Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Internal-authority SVR	0.424 fraction0.347–0.499 · scenario-percentile-bootstrap · level not stated	Conditions, limitations and source model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 01683Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
External-attack SVR	0.242 fraction0.184–0.305 · scenario-percentile-bootstrap · level not stated	Conditions, limitations and source model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 01684Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Internal minus external SVR	0.182 fraction-difference0.083–0.278 · scenario-percentile-bootstrap · level not stated	Conditions, limitations and source model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 01685Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Prompt reminder∆ASR	-0.107 fraction-difference-0.132–-0.083 · scenario-percentile-bootstrap · level not stated	Conditions, limitations and source model mean over 12 headline models; reminder deltas over9paired models Scenario-level composition uncertainty; confidence level not inferred from elsewhere in paper. Original source ↗ · Table17 PDFp18; Appendix H; text lines 01689Original capture 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828

Task and measurement

Benchmark protocol profile

Task / subject: The benchmark executes 155 sandbox scenarios spanning non-adversarial failures, internal-authority compromise and external attacks. Twelve models are run three times, totaling 5,580 executions. Repeats are averaged within scenario before averaging across eligible scenarios. ASR uses 105 attack-labeled scenarios; the other headline metrics use 155.
Measurement target: Can execution traces identify safety violations that final outcomes miss?
Interpretation boundary: The constructed distribution does not estimate natural incident prevalence. Appendix H bootstrap intervals describe scenario composition; the retained passage does not explicitly state their confidence level.

Measurement dossier · NUWA reading

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct: Can execution traces identify safety violations that final outcomes miss?
Task population: The benchmark executes 155 sandbox scenarios spanning non-adversarial failures, internal-authority compromise and external attacks. Twelve models are run three times, totaling 5,580 executions. Repeats are averaged within scenario before averaging across eligible scenarios. ASR uses 105 attack-labeled scenarios; the other headline metrics use 155.
Scoring and failure modes: TSR is utility-pass rate; SVR is endpoint safety-failure rate; ASR is that failure rate on attack-labeled scenarios; STCR requires task success without endpoint or latent violations. Latent violations are prohibited intermediate trace events. ASR uses 105 attack-labeled scenarios; the other headline metrics use 155. Each scenario averages three repeats.
System conditions: Source-specific interfaces and constructed task conditions; this reading does not establish matched comparability with other protocols.
Statistical meaning: Retain per-metric population, repeats and interval definitions from the record evidence table. Missing event numerators remain unknown.
Validity and interpretation: The constructed distribution does not estimate natural incident prevalence. Appendix H bootstrap intervals describe scenario composition; the retained passage does not explicitly state their confidence level.
Appropriate research use: NUWA recommendation: Use the joint metrics and trace locations to compare intervention points within this source. Do not rank these values against unrelated cyber benchmarks.

TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents · §3.6; §5–6; Tables 2,15,17; limitations; text lines 00483Original resource captured 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents · §3.2 illustrative case-dispute scenario; Table 24Original resource captured 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828

Original source charts and materials

Read the original protocol ↗

Selected original material has a measurement profile. Any numeric transcription remains in the private record evidence table; there is no approved chart or imported figure in this batch.

References

[1] TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents

Search NUWA Index

TraceProbe

Evidence update · 2026-10-28

A source-described sandbox case

Related reading

Task and measurement

Related research questions

Cybersecurity / Agent compromise through untrusted input →

References