RESEARCH RECORD

Research Note: Our scheming precursor evals had limited predictive power for our in-context scheming evals

· Apollo Research

预测效果随任务难度变化，困难版本可能误导。作者建议回测，并在可行时直接测量目标能力。容易到中等难度的预期顺序分别在75.9%的前置任务和55.3%的欺骗任务中成立；反向顺序分别占8.3%和7.9%。与通用智能分数的相关系数分别为0.816和0.412，原作者仅报告前者达到统计显著。

研究问题

前置能力测试能否可靠预警后续欺骗能力？

回顾比较不同难度的前置测试与上下文欺骗测试。

预测效果随任务难度变化，困难版本可能误导。

作者建议回测，并在可行时直接测量目标能力。

容易到中等难度的预期顺序分别在75.9%的前置任务和55.3%的欺骗任务中成立；反向顺序分别占8.3%和7.9%。

与通用智能分数的相关系数分别为0.816和0.412，原作者仅报告前者达到统计显著。

这是小规模研究笔记；前置测试和目标测试都存在效度限制。

系统	测量	原值	条件与位置
—	容易到中等难度符合预期顺序	75.9%%	Task proportion across precursor evaluations; underlying task denominator not disclosed in prose; 22 models and 100 runs per evaluation/model are separate countsL80
—	容易到中等难度顺序倒置	8.3%%	Task proportion across precursor evaluations; underlying task denominator not disclosed in prose; 22 models and 100 runs per evaluation/model are separate countsL80
—	容易到中等难度符合预期顺序	55.3%%	Task proportion across in-context scheming evaluations; underlying task denominator not disclosed in prose; 22 models and 100 runs per evaluation/model are separate countsL80
—	容易到中等难度顺序倒置	7.9%%	Task proportion across in-context scheming evaluations; underlying task denominator not disclosed in prose; 22 models and 100 runs per evaluation/model are separate countsL80
—	与通用智能分数的相关系数	r=0.816r	Precursors; statistically significant per authors; general intelligence scores from Artificial Analysis; not a probabilityL85-L88
—	与通用智能分数的相关系数	r=0.412r	Scheming; not statistically significant per authors; general intelligence scores from Artificial Analysis; not a probabilityL85-L88
—	评测模型数量	22 modelscount	Each evaluation run 100 times per model; number of models differs from task/run countL72

阅读范围与复核

Selected substantive passages of original institutional article; linked full paper, data and figures not independently audited

模型辅助整理，独立证据复核待完成。