WhitzardNUWA Lab ↗
NUWANUWA INDEX

Search NUWA Index

Research · Benchmarks · Findings · Perspectives

中文
← NUWA Index

Evaluation dossier

SafeAgentBench

Can embodied planners recognize hazardous instructions while completing safe tasks?

Affiliation not indexed in this review / Source capture · fc3f276d7f / Behavior

Sources and provenance ↓ Read the research narrative →

CROSS-REPORT MONITORING

PUBLISHED EVIDENCE

SafeAgentBench · Rejection ratio

Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.

180 observations · 0 dated

● Matched◇ Configuration differs□ Some conditions undisclosed
Rejection ratio (ratio) GPT-4 / LLM-Planner [6] Llama3-8B / MLDT [6] Llama3-8B / Lota-Bench [6] Llama3-8B / MLDT [5] GPT-4 / ReAct [6] GPT-4 / LLM-Planner [5] Llama3-8B / MLDT [2] GPT-4 / ReAct [5] GPT-4 / ReAct [2] GPT-4 / LLM-Planner [2] Llama3-8B / Lota-Bench [5] GPT-4 / ProgPrompt [5] 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.63 ratio 0.63 ratio 0.56 ratio 0.55 ratio 0.48 ratio 0.4 ratio 0.36 ratio 0.35 ratio 0.32 ratio 0.31 ratio 0.31 ratio 0.28 ratio
  1. Detailed hazardous tasks
  2. Abstract hazardous tasks
  3. Abstract tasks / L1
  4. Abstract tasks / L2
  5. Abstract tasks / L3
  6. Abstract tasks / L4

180 observations lack a verified model release date. They remain in Data and Systems.

Select a point or table row to inspect its conditions and provenance.

180 observations

Measurements and source conditions
System / modelResultCondition statusTask / conditionsEvaluator / reportDates
GPT-4 / LLM-Planner0.63 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / MLDT0.63 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / Lota-Bench0.56 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / MLDT0.55 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / ReAct0.48 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / LLM-Planner0.4 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / MLDT0.36 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / ReAct0.35 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / ReAct0.32 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / LLM-Planner0.31 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / Lota-Bench0.31 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / ProgPrompt0.28 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / ProgPrompt0.27 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / ProgPrompt0.26 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / Lota-Bench0.24 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / ReAct0.24 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / ProgPrompt0.2 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / MLDT0.2 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / MLDT0.2 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / ProgPrompt0.19 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / ReAct0.19 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / MLDT0.17 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / Lota-Bench0.16 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / MLDT0.15 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / ProgPrompt0.15 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / MLDT0.13 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / ProgPrompt0.13 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / ProgPrompt0.13 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / LLM-Planner0.12 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / Lota-Bench0.11 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / ReAct0.1 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / ProgPrompt0.1 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / LLM-Planner0.1 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / Lota-Bench0.1 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / ProgPrompt0.09 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / ProgPrompt0.09 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / ProgPrompt0.09 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / MLDT0.08 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / ProgPrompt0.08 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / ProgPrompt0.07 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / ProgPrompt0.07 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / ProgPrompt0.07 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / ProgPrompt0.07 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / ProgPrompt0.06 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / MLDT0.05 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / Lota-Bench0.05 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / MLDT0.05 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / ProgPrompt0.03 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / Lota-Bench0.03 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / MLDT0.03 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / MLDT0.03 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / ProgPrompt0.03 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / MLDT0.02 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / Lota-Bench0.02 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / MLDT0.02 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / MLDT0.01 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / Lota-Bench0.01 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / MLDT0.01 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / Lota-Bench0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / Lota-Bench0 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / LLM-Planner0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / CoELA0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)CoELA adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / CoELA0 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)CoELA adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / MAP0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)MAP adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / MAP0 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)MAP adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / PCA-EVAL0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / PCA-EVAL0 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / KARMA0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)KARMA adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / KARMA0 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)KARMA adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / LLM-Planner0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / LLM-Planner0 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / CoELA0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)CoELA adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / CoELA0 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)CoELA adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / MLDT0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / ProgPrompt0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / ReAct0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / ReAct0 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / PCA-EVAL0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / PCA-EVAL0 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / Lota-Bench0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / LLM-Planner0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / LLM-Planner0 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / CoELA0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)CoELA adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / CoELA0 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)CoELA adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / MLDT0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / ProgPrompt0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / ReAct0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / ReAct0 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / PCA-EVAL0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / PCA-EVAL0 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / Lota-Bench0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / LLM-Planner0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / LLM-Planner0 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / Co-LLM-Agent0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)Co-LLM-Agent adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / Co-LLM-Agent0 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)Co-LLM-Agent adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / MLDT0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / ProgPrompt0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / ReAct0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / ReAct0 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / PCA-EVAL0 ratioRejection ratioIncompleteDetailed hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / PCA-EVAL0 ratioRejection ratioIncompleteAbstract hazardous taskstemperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / Lota-Bench0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / Lota-Bench0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / Lota-Bench0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / Lota-Bench0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / CoELA0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)CoELA adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / CoELA0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)CoELA adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / CoELA0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)CoELA adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / CoELA0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)CoELA adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / MAP0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)MAP adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / MAP0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)MAP adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / MAP0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)MAP adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / MAP0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)MAP adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / PCA-EVAL0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / PCA-EVAL0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / PCA-EVAL0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / PCA-EVAL0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / KARMA0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)KARMA adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / KARMA0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)KARMA adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / KARMA0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)KARMA adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
GPT-4 / KARMA0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)KARMA adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / Lota-Bench0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / LLM-Planner0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / LLM-Planner0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / LLM-Planner0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / LLM-Planner0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / Co-LLM-Agent0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)Co-LLM-Agent adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / Co-LLM-Agent0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)Co-LLM-Agent adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / Co-LLM-Agent0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)Co-LLM-Agent adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / Co-LLM-Agent0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)Co-LLM-Agent adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / ProgPrompt0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)ProgPrompt adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / ReAct0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / ReAct0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / ReAct0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / ReAct0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / PCA-EVAL0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / PCA-EVAL0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / PCA-EVAL0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Llama3-8B / PCA-EVAL0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / Lota-Bench0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / Lota-Bench0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / LLM-Planner0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / LLM-Planner0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / LLM-Planner0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / LLM-Planner0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / Co-LLM-Agent0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)Co-LLM-Agent adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / Co-LLM-Agent0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)Co-LLM-Agent adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / Co-LLM-Agent0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)Co-LLM-Agent adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / Co-LLM-Agent0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)Co-LLM-Agent adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / MLDT0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / MLDT0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / ReAct0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / ReAct0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / ReAct0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / ReAct0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / PCA-EVAL0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / PCA-EVAL0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / PCA-EVAL0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
Qwen2-7B / PCA-EVAL0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / Lota-Bench0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / Lota-Bench0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / Lota-Bench0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)Lota-Bench adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / LLM-Planner0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / LLM-Planner0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / LLM-Planner0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / LLM-Planner0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)LLM-Planner adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / Co-LLM-Agent0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)Co-LLM-Agent adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / Co-LLM-Agent0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)Co-LLM-Agent adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / Co-LLM-Agent0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)Co-LLM-Agent adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / Co-LLM-Agent0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)Co-LLM-Agent adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / MLDT0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / MLDT0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)MLDT adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / ReAct0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / ReAct0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / ReAct0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / ReAct0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)ReAct adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / PCA-EVAL0 ratioRejection ratioIncompleteAbstract tasks / L1temperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / PCA-EVAL0 ratioRejection ratioIncompleteAbstract tasks / L2temperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / PCA-EVAL0 ratioRejection ratioIncompleteAbstract tasks / L3temperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17
DeepSeek-V2.5 / PCA-EVAL0 ratioRejection ratioIncompleteAbstract tasks / L4temperature=0; max_tokens=4096 (Appendix C.1)PCA-EVAL adapted to SafeAgentEnvNot disclosedSafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM AgentsRelease —Report 2024-12-17

Evaluated by: Not disclosed
Reported by: SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents

Task and measurement

Benchmark protocol profile

Task / subject
Executable household simulations distinguish detailed, abstract, and long-horizon tasks; execution and semantic judgments provide complementary measurements.
Measurement target
Can embodied planners recognize hazardous instructions while completing safe tasks?
Scoring / outcome
Rejection ratio · Rejection ratio
Interpretation boundary
The low-level controller maps high-level actions to simulator APIs; this is not validation of a physical robot or learned motor controller. Hazardous examples are not reproduced here.

A reported evaluation

The source evaluates GPT-4 / Lota-Bench · Detailed hazardous tasks.

Run configuration: Lota-Bench adapted to SafeAgentEnv · temperature=0; max_tokens=4096 (Appendix C.1).

Reported measurement: Rejection ratio · 0 ratio.

Rejection ratio Not disclosed

The revised paper reports weak hazard awareness across planning approaches, despite differences in task completion. Changing the underlying LLM alone does not reliably solve the problem.

Measurement dossier · NUWA reading

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct
Safety-aware embodied task planning, separating refusal, risky execution and safe completion.
Task population
SafeAgentBench contains 750 simulated embodied tasks, including 450 hazardous and 300 paired safe tasks. Detailed, abstract and long-horizon tasks test different planning demands.
Scoring and failure modes
Detailed tasks report rejection, goal-based/LLM-based risk ratios and execution; abstract tasks report rejection and risk ratios; long-horizon tasks distinguish completed-safe, completed-unsafe and incomplete. Source ratios are retained on their original 0–1 scale.
System conditions
Baseline agents are adapted to SafeAgentEnv. Appendix C.1 specifies temperature 0 and max_tokens 4096; Appendix C.2 lists 17 high-level actions. GPT-4 main results and three open-model appendix results are separate conditions.
Statistical meaning
Not established by the retained material
Validity and interpretation
Simulator task performance does not establish real-world physical incident frequency. Task type, evaluator and agent design change the meaning of each reported ratio.
Appropriate research use
Not established by the retained material

Evidence across reports

The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.

SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents

SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents

Actual evaluators: Not disclosed

540 result records · 32 reported system names

Protocols and comparison decisions (0)
Original source charts and materials

Read the original protocol ↗

The low-level controller maps high-level actions to simulator APIs; this is not validation of a physical robot or learned motor controller. Hazardous examples are not reproduced here.

Related research questions

References

Numerical results are reported by the cited authors.

  1. [1] SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents. 2024-12-17 ; #S4.T2; row GPT-4 / Lota-Bench; condition Detailed hazardous tasks; column Rejection ratio; #S4.T2; row GPT-4 / Lota-Bench; condition Detailed hazardous tasks; column Risk ratio (goal evaluator)