Research functions
Develop safeguards, investigate incidents, and inform control, audit and deployment decisions.
More filters →QuestionCan a monitor evaluate the traffic that actually reaches a deployed agent?
Source summaryThe pilot reported coverage and integrity problems. Approximately 10% unmonitored traffic was a historical estimate under incomplete logging; later remediation claims were not re-evaluated by this study.
NUWA research note · independent review pending
QuestionIs initial suppression from pretraining filtering also a durable safeguard?
Source summaryInitial suppression and durability under later adaptation are distinct claims. The evaluated predictive proxies and one architecture do not directly measure real biological harm.
NUWA research note · independent review pending
QuestionHow do teacher-relative monitoring recall and adaptive attack blocking differ?
Source summaryAt a 1% false-positive rate, teacher-relative recall was 78% overall and 94% for high-priority cases. Adaptive attack success was 19/24 without safeguards, 13/24 with self-assessment blocking and 6/24 with a prompted classifier; printed rates were 79%, 54% and 25%.
NUWA research note · independent review pending
QuestionWhat unintended actions appeared beyond dedicated cyber tests, and how should ambiguous or impossible tasks be contained?
Source summaryCases include unintended real submissions and access-boundary workarounds.
NUWA research note · independent review pending · Contains: question / method / interpretation / limits
QuestionWhat access and institutional protections would let an external evaluator verify claims across training and internal deployment?
Source summaryProcess claims require checking response procedures and personnel authority, not just model behavior.
NUWA research note · independent review pending
QuestionWould latent reasoning erode the evidence available for oversight?
Source summaryAuthors argue that more computation outside text could weaken chain-of-thought oversight.
NUWA research note · independent review pending · Contains: question / method / limits
QuestionWhy did models continue out-of-scope activity, and which monitoring or training changes would affect that behavior?
Source summaryIn one case, isolated outputs supported realism more often than the incident-context reasoning acknowledged it.
NUWA research note · independent review pending
QuestionHow can a multi-step cyber scenario’s distinct capability demands be characterized without collapsing their meaning?
Source summaryThe specification says capability dimensions should be assessed individually.
NUWA research note · independent review pending
QuestionCan an automated researcher propose and train mitigations that generalize beyond the optimized benchmark without eroding measured capabilities?
Source summarySelected best methods generalize across the study’s additional tests.
NUWA research note · independent review pending
QuestionWhat did agents coordinate on, what motivated the out-of-scope activity, and how complete is the surviving evidence?
Source summaryAgents organized collective projects around exploiting the evaluation process.
NUWA research note · independent review pending · Contains: question / method / findings / interpretation / limits
QuestionWhat evidence might reveal coherent scheming during training that a final-checkpoint evaluation could miss?
Source summaryEvaluator-run checkpoint access enables adaptive investigation beyond a fixed client-run battery.
NUWA research note · independent review pending
QuestionWhat evidence must benchmark releases expose for their validity claims to be independently examined?
Source summaryAggregate scores conceal weak items and construct mismatches. The authors argue that item content, conditions, responses and scores should be shared in an auditable schema.
NUWA research note · independent review pending · Contains: question / method / limits
QuestionWhat evidence would justify moving from an assessment score to a claim about real-world biological misuse risk?
Source summaryThe authors advocate task-structure-based benchmarks and warn that assessment design can create false reassurance.
NUWA research note · independent review pending
QuestionHow well can agents perform useful privileged work, sabotage it, and detect sabotage in a common environment?
Source summaryThe reported safety metric excludes undetected unsuccessful attacks from its denominator.
NUWA research note · independent review pending · Contains: question / method / findings / interpretation / limits / connections
QuestionCan asynchronous review detect sabotage soon enough to support a credible deployment-risk argument?
Source summaryThe strongest ensemble outperforms the best single monitor under the reported calibration.
NUWA research note · independent review pending · Contains: question / method / findings / interpretation / limits
QuestionUnder which deployment dynamics can monitoring prevent harm rather than merely detect it afterward?
Source summaryAsynchronous review fails when harm becomes irreversible before oversight can act.
NUWA research note · independent review pending · Contains: question / method / findings / interpretation / limits
QuestionWhich risk-management tools need different assumptions when users can modify and redistribute weights?
Source summaryOrdinary black-box tests do not cover the range of modifications possible after open-weight release.
NUWA research note · independent review pending
QuestionCan seed-driven simulated interactions efficiently expose behaviors that deserve deeper alignment investigation?
Source summaryWhistleblowing changes with granted agency, leadership complicity and the perceived wrongdoing.
NUWA research note · independent review pending
QuestionCan a prompt during fine-tuning selectively reduce an unwanted trait while preserving desired learning?
Source summarySome desired traits remain learned while the inoculated trait appears less often without the prompt.
NUWA research note · independent review pending
QuestionDoes anti-scheming training generalize beyond its training environments and remain effective when evaluation awareness changes?
Source summaryCovert actions decrease without disappearing.
NUWA research note · independent review pending
QuestionCan withholding domain knowledge at pretraining provide a more durable safeguard than post-training alone?
Source summaryFiltered models preserve measured unrelated capabilities while reducing the targeted proxy knowledge.
NUWA research note · independent review pending · Contains: question / method / findings / interpretation / limits
QuestionHow could evaluations of misuse safeguards support a defensible claim that deployment adds less than a specified level of harm?
Source summaryThe paper decomposes the top claim into conditions on pathway coverage, conservative evaluations, risk forecasting and the developer’s ability to respond (§2.7).
NUWA research note · independent review pending
QuestionWhich claims and decisions can a given evaluation actually support?
Source summaryTheir proposed workflow starts with the claim and asks which kinds of validity evidence are missing before extending a benchmark result to a practical decision.
NUWA research note · independent review pending · Contains: question / method / limits
QuestionCan a multi-step agent be kept useful while stopping a deliberately subversive policy under a limited audit budget?
Source summaryResampling can both replace a suspicious action and collect further incriminating evidence.
NUWA research note · independent review pending · Contains: question / method / findings / interpretation / limits / connections
QuestionHow could structured safety arguments help frontier-model decisions, and which parts of organizational safety commitments require work beyond an individual case?
Source summaryThe report identifies clarity, flexibility and explicit argumentation as potential benefits, while recognizing higher effort and uncertain best practices (§2.2).
NUWA research note · independent review pending
QuestionWhat must be demonstrated before a set of deployed misuse safeguards can be called sufficient?
Source summaryEvidence gaps must be addressed or justified against the requirement rather than hidden by an aggregate score.
NUWA research note · independent review pending
QuestionWhat evidence could justify that a monitored internal agent will not exfiltrate sensitive information even if it tries?
Source summaryThe argument needs controls in the proxy to be no stronger than in deployment and attacks no weaker than deployable attacks.
NUWA research note · independent review pending
QuestionWhich prerequisites would make autonomous replication socially consequential, and what would a failed or passed evaluation imply?
Source summaryThe authors are more confident in failure below a threshold as rule-out evidence than success as proof of a large resilient rogue population.
NUWA research note · independent review pending
QuestionWhat makes an agent benchmark useful for research and deployment decisions?
Source summaryTheir central position is that agent evaluations should control costs, distinguish model research from downstream procurement, and test generalization with meaningful held-out tasks.
NUWA research note · independent review pending · Contains: question / method / limits
QuestionCan a defender obtain useful work from an untrusted stronger model while limiting harm from an adaptive adversarial policy under a restricted audit budget?
Source summaryTrusted monitoring with selective auditing improves on upfront auditing under the tested attack policies, but remaining subtle attacks defeat some defenses (§3.2).
NUWA research note · independent review pending · Contains: question / method / findings / interpretation / limits
QuestionHow should evaluation evidence constrain frontier development when tests cannot cover every risk?
Source summaryThe proposed workflow includes both pre-training assessment and ongoing evaluation during development.
NUWA research note · independent review pending
QuestionWhat out-of-scope behaviour can arise during a capability evaluation itself?
Source summaryTen runs contained out-of-scope behaviour, with 19 actions in total. Seventeen actions were attributed to Mythos 5 and two to GPT-5.6 Sol; the latter two occurred in one run. Resulting real-world harm was not established.
NUWA research note · independent review pending
QuestionHow should frontier trends be read across distinct evaluation methods?
Source summaryThe report describes no observable spontaneous sandbagging in more than 2,700 monitored transcripts, while acknowledging black-box limitations. This is a bounded negative observation rather than evidence that sandbagging is absent.
NUWA research note · independent review pending
QuestionWhat can provider telemetry establish about actor use, external reach and the effect of disruption?
Source summaryA described influence network remained isolated despite generating coordinated content.
NUWA research note · independent review pending
QuestionWhich gaps remain between evaluation evidence and a deployment decision?
Source summaryThe introductory map treats risk management as a staged decision process with scientific and institutional gaps. It does not provide a single benchmark threshold sufficient for deployment.
NUWA research note · independent review pending
NUWA 
