We need 3rd party Training-Run Evaluations

Research question

What evidence might reveal coherent scheming during training that a final-checkpoint evaluation could miss?

Design and assumptions

The essay analyzes covert behavior and obfuscated cognition, then distinguishes intermediate-checkpoint access from deeper access to data, rewards, environments and interventions.

Source findings

Evaluator-run checkpoint access enables adaptive investigation beyond a fixed client-run battery.

The authors separate diffuse and concentrated IP concerns and propose gradually increasing access; they do not report a completed general TRE standard.

NUWA interpretation

NUWA interpretation: a negative final test cannot validate a claim about the whole training process without evidence about that process.

Limits and contrary evidence

This is an institutional argument under explicit uncertainty about when scheming becomes undetectable; it is not empirical proof that TREs detect all scheming.

Research connections

NUWA connection: embedded-evaluators extends the access argument to continuing deployment and governance; anti-scheming-training supplies a concrete evaluation-awareness concern.

Research and decision relevance

NUWA recommendation: bind each training-safety claim to required access, checkpoint timing and denied information before requesting external assurance.