RESEARCH RECORD

Research Note: Our scheming precursor evals had limited predictive power for our in-context scheming evals

· Apollo Research

Predictive performance varied with task difficulty; hard precursors could mislead.The authors recommend back-testing and direct measurement where possible.Easy-to-medium expected ordering holds for75.9% of precursor tasks and55.3% of scheming tasks; reverse ordering occurs in8.3% and7.9%.Correlations with general intelligence scores are r=.816 for precursors and r=.412 for scheming; only the former is reported statistically significant.

Research question

Do precursor tests reliably warn of later scheming capabilities?

Design & assumptions

Retrospective comparison of precursor and in-context tests across difficulty levels.

Findings

Predictive performance varied with task difficulty; hard precursors could mislead.

The authors recommend back-testing and direct measurement where possible.

Easy-to-medium expected ordering holds for75.9% of precursor tasks and55.3% of scheming tasks; reverse ordering occurs in8.3% and7.9%.

Correlations with general intelligence scores are r=.816 for precursors and r=.412 for scheming; only the former is reported statistically significant.

Limits & counterevidence

Small, less-rigorous research note; both target and precursor tests have validity limitations.

Source evidence

System	Measure	Reported value	Conditions & location
—	Expected easy→medium ordering	75.9%%	Task proportion across precursor evaluations; underlying task denominator not disclosed in prose; 22 models and 100 runs per evaluation/model are separate countsL80
—	Reverse easy→medium ordering	8.3%%	Task proportion across precursor evaluations; underlying task denominator not disclosed in prose; 22 models and 100 runs per evaluation/model are separate countsL80
—	Expected easy→medium ordering	55.3%%	Task proportion across in-context scheming evaluations; underlying task denominator not disclosed in prose; 22 models and 100 runs per evaluation/model are separate countsL80
—	Reverse easy→medium ordering	7.9%%	Task proportion across in-context scheming evaluations; underlying task denominator not disclosed in prose; 22 models and 100 runs per evaluation/model are separate countsL80
—	Correlation with general intelligence scores	r=0.816r	Precursors; statistically significant per authors; general intelligence scores from Artificial Analysis; not a probabilityL85-L88
—	Correlation with general intelligence scores	r=0.412r	Scheming; not statistically significant per authors; general intelligence scores from Artificial Analysis; not a probabilityL85-L88
—	Evaluated models	22 modelscount	Each evaluation run 100 times per model; number of models differs from task/run countL72

Reading scope & review

Selected substantive passages of original institutional article; linked full paper, data and figures not independently audited

Model-assisted reading; independent evidence review pending.

Check the original source ↗