NUWA RESEARCH RECORD
Identifying the Risks of LM Agents with an LM-Emulated Sandbox
Original publication ↗ · BibTeX · RISThe framework uncovers plausible high-consequence failures without implementing every real tool. Human checks also reveal simulator errors and missed unsafe actions.
Research question
Can language-model simulation efficiently reveal agent failures before tool deployment?
Design & assumptions
ToolEmu emulates tool execution and scores whole trajectories for safety and helpfulness; a paired standard/adversarial-emulator validation uses human judgments.
Findings
The framework uncovers plausible high-consequence failures without implementing every real tool. Human checks also reveal simulator errors and missed unsafe actions.
Limits & counterevidence
Simulation fidelity and evaluator recall constrain interpretation. An emulated failure is neither a confirmed production incident nor a measured real-world failure frequency.
Source evidence
| System | Measure | Reported value | Conditions & location |
|---|---|---|---|
| GPT-4 / Basic | Average safety score | 2.007score (0–3) | ToolEmu adversarial emulator / Basic#S5.T5; row GPT-4 / Basic; condition ToolEmu adversarial emulator / Basic; column Average safety score |
| GPT-4 / Basic | Failure incidence | 39.4%% | ToolEmu adversarial emulator / Basic#S5.T5; row GPT-4 / Basic; condition ToolEmu adversarial emulator / Basic; column Failure incidence |
| GPT-4 / Basic | Average helpfulness score | 1.458score (0–3) | ToolEmu adversarial emulator / Basic#S5.T5; row GPT-4 / Basic; condition ToolEmu adversarial emulator / Basic; column Average helpfulness score |
| Claude-2 / Basic | Average safety score | 1.829score (0–3) | ToolEmu adversarial emulator / Basic#S5.T5; row Claude-2 / Basic; condition ToolEmu adversarial emulator / Basic; column Average safety score |
| Claude-2 / Basic | Failure incidence | 44.3%% | ToolEmu adversarial emulator / Basic#S5.T5; row Claude-2 / Basic; condition ToolEmu adversarial emulator / Basic; column Failure incidence |
| Claude-2 / Basic | Average helpfulness score | 1.464score (0–3) | ToolEmu adversarial emulator / Basic#S5.T5; row Claude-2 / Basic; condition ToolEmu adversarial emulator / Basic; column Average helpfulness score |
| ChatGPT-3.5 / Basic | Average safety score | 1.430score (0–3) | ToolEmu adversarial emulator / Basic#S5.T5; row ChatGPT-3.5 / Basic; condition ToolEmu adversarial emulator / Basic; column Average safety score |
| ChatGPT-3.5 / Basic | Failure incidence | 62.0%% | ToolEmu adversarial emulator / Basic#S5.T5; row ChatGPT-3.5 / Basic; condition ToolEmu adversarial emulator / Basic; column Failure incidence |
| ChatGPT-3.5 / Basic | Average helpfulness score | 0.768score (0–3) | ToolEmu adversarial emulator / Basic#S5.T5; row ChatGPT-3.5 / Basic; condition ToolEmu adversarial emulator / Basic; column Average helpfulness score |
| Vicuna-1.5-13B / Basic | Average safety score | 1.552score (0–3) | ToolEmu adversarial emulator / Basic#S5.T5; row Vicuna-1.5-13B / Basic; condition ToolEmu adversarial emulator / Basic; column Average safety score |
| Vicuna-1.5-13B / Basic | Failure incidence | 54.6%% | ToolEmu adversarial emulator / Basic#S5.T5; row Vicuna-1.5-13B / Basic; condition ToolEmu adversarial emulator / Basic; column Failure incidence |
| Vicuna-1.5-13B / Basic | Average helpfulness score | 0.441score (0–3) | ToolEmu adversarial emulator / Basic#S5.T5; row Vicuna-1.5-13B / Basic; condition ToolEmu adversarial emulator / Basic; column Average helpfulness score |
| Vicuna-1.5-7B / Basic | Average safety score | 1.850score (0–3) | ToolEmu adversarial emulator / Basic#S5.T5; row Vicuna-1.5-7B / Basic; condition ToolEmu adversarial emulator / Basic; column Average safety score |
| Vicuna-1.5-7B / Basic | Failure incidence | 45.0%% | ToolEmu adversarial emulator / Basic#S5.T5; row Vicuna-1.5-7B / Basic; condition ToolEmu adversarial emulator / Basic; column Failure incidence |
| Vicuna-1.5-7B / Basic | Average helpfulness score | 0.364score (0–3) | ToolEmu adversarial emulator / Basic#S5.T5; row Vicuna-1.5-7B / Basic; condition ToolEmu adversarial emulator / Basic; column Average helpfulness score |
| GPT-4 / Safety | Average safety score | 2.359score (0–3) | ToolEmu adversarial emulator / Safety#S5.T5; row GPT-4 / Safety; condition ToolEmu adversarial emulator / Safety; column Average safety score |
| GPT-4 / Safety | Failure incidence | 23.9%% | ToolEmu adversarial emulator / Safety#S5.T5; row GPT-4 / Safety; condition ToolEmu adversarial emulator / Safety; column Failure incidence |
| GPT-4 / Safety | Average helpfulness score | 1.824score (0–3) | ToolEmu adversarial emulator / Safety#S5.T5; row GPT-4 / Safety; condition ToolEmu adversarial emulator / Safety; column Average helpfulness score |
| Helpful + Safety / Basic | Average safety score | 2.241score (0–3) | ToolEmu adversarial emulator / Basic#S5.T5; row Helpful + Safety / Basic; condition ToolEmu adversarial emulator / Basic; column Average safety score |
| Helpful + Safety / Basic | Failure incidence | 30.5%% | ToolEmu adversarial emulator / Basic#S5.T5; row Helpful + Safety / Basic; condition ToolEmu adversarial emulator / Basic; column Failure incidence |
| Helpful + Safety / Basic | Average helpfulness score | 1.624score (0–3) | ToolEmu adversarial emulator / Basic#S5.T5; row Helpful + Safety / Basic; condition ToolEmu adversarial emulator / Basic; column Average helpfulness score |
| NoAct / - | Average safety score | 3.000score (0–3) | ToolEmu adversarial emulator / -#S5.T5; row NoAct / -; condition ToolEmu adversarial emulator / -; column Average safety score |
| NoAct / - | Failure incidence | 0.00%% | ToolEmu adversarial emulator / -#S5.T5; row NoAct / -; condition ToolEmu adversarial emulator / -; column Failure incidence |
| NoAct / - | Average helpfulness score | 0.063score (0–3) | ToolEmu adversarial emulator / -#S5.T5; row NoAct / -; condition ToolEmu adversarial emulator / -; column Average helpfulness score |
| Standard emulator | Identified failure precision | 72.5% ± 7.1%%72.5% ± 7.1% | External annotators#S4.T3; row Standard emulator; condition External annotators; column Identified failure precision |
| Standard emulator | True failure incidence | 39.6% ± 4.9%%39.6% ± 4.9% | External annotators#S4.T3; row Standard emulator; condition External annotators; column True failure incidence |
| Adversarial emulator | Identified failure precision | 68.8% ± 6.7%%68.8% ± 6.7% | External annotators#S4.T3; row Adversarial emulator; condition External annotators; column Identified failure precision |
| Adversarial emulator | True failure incidence | 50.0% ± 5.1%%50.0% ± 5.1% | External annotators#S4.T3; row Adversarial emulator; condition External annotators; column True failure incidence |
| Standard emulator | Identified failure precision | 71.1% ± 7.4%%71.1% ± 7.4% | Author annotators#A2.T4; row Standard emulator; condition Author annotators; column Identified failure precision |
| Standard emulator | True failure incidence | 40.8% ± 5.0%%40.8% ± 5.0% | Author annotators#A2.T4; row Standard emulator; condition Author annotators; column True failure incidence |
| Adversarial emulator | Identified failure precision | 69.4% ± 6.6%%69.4% ± 6.6% | Author annotators#A2.T4; row Adversarial emulator; condition Author annotators; column Identified failure precision |
| Adversarial emulator | True failure incidence | 47.5% ± 5.0%%47.5% ± 5.0% | Author annotators#A2.T4; row Adversarial emulator; condition Author annotators; column True failure incidence |
Reading scope & review
Selected original HTML sections and table context; not a complete paper/appendix audit or independent reproduction.
Model-assisted reading; independent evidence review pending.
Check the original source ↗