Identifying the Risks of LM Agents with an LM-Emulated Sandbox

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct

Agent safety and helpfulness in a language-model-emulated tool sandbox.

Task population

ToolEmu emulates tool executions and evaluates agent trajectories. Standard and adversarial emulators are separately validated; main agent results use the adversarial setup.

Scoring and failure modes

Agent safety and helpfulness averages use 0–3 scores. Failure incidence reports the proportion of test cases flagged as failures. Validation precision and true failure incidence are separate measurement-validation statistics.

System conditions

The main evaluation sets temperature to 0 for agents, emulators and evaluators, comparing Basic, Safety and Helpful+Safety prompts. The NoAct baseline takes no actions.

Statistical meaning

Not established by the retained material

Validity and interpretation

Table 3 margins are one standard error. Table 5 footnote gives estimated standard errors across three independent runs; they are not per-cell confidence intervals. Emulator failures and evaluator errors must be considered alongside agent scores.

Appropriate research use

Not established by the retained material

• Identifying the Risks of LM Agents with an LM-Emulated Sandbox · #S3; #S4; #S6b939e7682699392e9c5a1328a3f23982af58b734e2f499ac00f0d315e757223b