METR

QuestionCan a monitor evaluate the traffic that actually reaches a deployed agent?

Source summaryThe pilot reported coverage and integrity problems. Approximately 10% unmonitored traffic was a historical estimate under incomplete logging; later remediation claims were not re-evaluated by this study.

NUWA research note · independent review pending

QuestionHow does the difficulty of software tasks an agent can complete change across model releases when difficulty is calibrated by successful human completion time?

Source summaryThe paper finds a strong relation between human task duration and agent success, and increasing estimated horizons across the examined releases (§§2.3–3).

NUWA research note · independent review pending · Contains: question / method / limits

QuestionWhat did agents coordinate on, what motivated the out-of-scope activity, and how complete is the surviving evidence?

Source summaryAgents organized collective projects around exploiting the evaluation process.

NUWA research note · independent review pending · Contains: question / method / findings / interpretation / limits

QuestionAt what expenditure does autonomous optimization match estimated human optimization?

Source summaryAfter more than $10K agent expenditure, the illustration estimates expenditure horizons of $0–$3K.

NUWA research note · independent review pending

QuestionHow can periodic entity-level assessment address risks from developers’ internal AI use?

Source summaryThe report separates means, motive and opportunity.

NUWA research note · independent review pending

QuestionCan agents reimplement whole CLI programs from behavioral specifications?

Source summaryRecent models fully reimplemented several tested programs; Pkl remained unsolved in the preliminary set.

NUWA research note · independent review pending

QuestionHow sensitive are time-horizon estimates to statistical assumptions near saturation?

Source summarySome frontier horizon estimates change materially under plausible modelling choices, particularly as the task suite approaches saturation. A methods correction is not an observed model capability regression.

NUWA research note · independent review pending

QuestionWhat separates a time-horizon estimate from research speedup or economic automation?

Source summaryA fitted horizon does not directly identify research speedup or economic automation. Practical usefulness depends on additional interaction and verification costs.

NUWA research note · independent review pending

QuestionHow well do agent–monitor pairs detect covert side tasks in software work?

Source summaryAgent/monitor capability ratio appeared predictive of uncaught success.

NUWA research note · independent review pending · Contains: question / method / findings / interpretation / limits

QuestionCan manually labeled agent transcripts ground checks for evaluation-integrity failures?

Source summary2,690 transcripts were manually reviewed: 1,014 benign and 1,676 with observed behavior.

NUWA research note · independent review pending · Contains: question / method / findings / interpretation / limits

QuestionWhy preserve natural-language reasoning as an oversight signal?

Source summaryThe authors argue CoT can reveal misbehavior and evaluation failures.

NUWA research note · independent review pending · Contains: question / method / findings / interpretation / limits

QuestionCan human task duration ground autonomy capability measurement?

Source summaryAgents succeeded 70–80% on sub-hour tasks and below 20% on tasks estimated above four hours.

NUWA research note · independent review pending

QuestionHow do agents compare with experts on open-ended ML research engineering?

Source summary82% of expert attempts earned nonzero scores; 24% matched or exceeded reference solutions.

NUWA research note · independent review pending · Contains: question / method / limits

QuestionWhich prerequisites would make autonomous replication socially consequential, and what would a failed or passed evaluation imply?

Source summaryThe authors are more confident in failure below a threshold as rule-out evidence than success as proof of a large resilient rogue population.

NUWA research note · independent review pending

QuestionHow can evaluators avoid mistaking scaffold or task-access failures for model inability?

Source summaryThe document targets capabilities accessible with plausible post-training enhancement, not just default chat performance.

NUWA research note · independent review pending