NUWA RESEARCH RECORD
R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
Original publication ↗ · BibTeX · RISThe paper reports gaps in risk awareness and finds that fine-tuning can help more than straightforward prompting in its evaluated setup.
Research question
Can a model identify safety risks in an existing agent interaction trace?
Design & assumptions
Human-annotated multi-turn traces provide binary safety labels and risk descriptions; evaluation separates safety classification from explanation quality.
Findings
The paper reports gaps in risk awareness and finds that fine-tuning can help more than straightforward prompting in its evaluated setup.
Limits & counterevidence
This tests judging recorded trajectories rather than the agent’s ability to act safely online. Dataset size and curated risk coverage constrain transfer; version-specific counts must not be mixed.
Source evidence
| System | Measure | Reported value | Conditions & location |
|---|---|---|---|
| GPT-4o | F1 | 74.45% | Safety judgement and risk identification / All#S3.T1; row GPT-4o; column F1; condition All |
| GPT-4o | F1 | 72.19% | Safety judgement and risk identification / Intended attacks#S3.T1; row GPT-4o; column F1; condition Intended attacks |
| GPT-4o | Recall | 91.50% | Safety judgement and risk identification / Intended attacks#S3.T1; row GPT-4o; column Recall; condition Intended attacks |
| GPT-4o | Specificity | 42.06% | Safety judgement and risk identification / Intended attacks#S3.T1; row GPT-4o; column Specificity; condition Intended attacks |
| GPT-4o | Effectiveness | 93% | Safety judgement and risk identification / Intended attacks#S3.T1; row GPT-4o; column Effectiveness; condition Intended attacks |
| GPT-4o | F1 | 80.90% | Safety judgement and risk identification / Unintended risks#S3.T1; row GPT-4o; column F1; condition Unintended risks |
| GPT-4o | Recall | 72.00% | Safety judgement and risk identification / Unintended risks#S3.T1; row GPT-4o; column Recall; condition Unintended risks |
| GPT-4o | Specificity | 89.09% | Safety judgement and risk identification / Unintended risks#S3.T1; row GPT-4o; column Specificity; condition Unintended risks |
| GPT-4o | Effectiveness | 78% | Safety judgement and risk identification / Unintended risks#S3.T1; row GPT-4o; column Effectiveness; condition Unintended risks |
| ChatGPT | F1 | 44.96% | Safety judgement and risk identification / All#S3.T1; row ChatGPT; column F1; condition All |
| ChatGPT | F1 | 40.55% | Safety judgement and risk identification / Intended attacks#S3.T1; row ChatGPT; column F1; condition Intended attacks |
| ChatGPT | Recall | 37.00% | Safety judgement and risk identification / Intended attacks#S3.T1; row ChatGPT; column Recall; condition Intended attacks |
| ChatGPT | Specificity | 57.48% | Safety judgement and risk identification / Intended attacks#S3.T1; row ChatGPT; column Specificity; condition Intended attacks |
| ChatGPT | Effectiveness | 36.5% | Safety judgement and risk identification / Intended attacks#S3.T1; row ChatGPT; column Effectiveness; condition Intended attacks |
| ChatGPT | F1 | 55.63% | Safety judgement and risk identification / Unintended risks#S3.T1; row ChatGPT; column F1; condition Unintended risks |
| ChatGPT | Recall | 42.00% | Safety judgement and risk identification / Unintended risks#S3.T1; row ChatGPT; column Recall; condition Unintended risks |
| ChatGPT | Specificity | 83.64% | Safety judgement and risk identification / Unintended risks#S3.T1; row ChatGPT; column Specificity; condition Unintended risks |
| ChatGPT | Effectiveness | 41.5% | Safety judgement and risk identification / Unintended risks#S3.T1; row ChatGPT; column Effectiveness; condition Unintended risks |
| Meta-Llama-3-8B-Instruct | F1 | 61.01% | Safety judgement and risk identification / All#S3.T1; row Meta-Llama-3-8B-Instruct; column F1; condition All |
| Meta-Llama-3-8B-Instruct | F1 | 65.68% | Safety judgement and risk identification / Intended attacks#S3.T1; row Meta-Llama-3-8B-Instruct; column F1; condition Intended attacks |
| Meta-Llama-3-8B-Instruct | Recall | 66.50% | Safety judgement and risk identification / Intended attacks#S3.T1; row Meta-Llama-3-8B-Instruct; column Recall; condition Intended attacks |
| Meta-Llama-3-8B-Instruct | Specificity | 66.36% | Safety judgement and risk identification / Intended attacks#S3.T1; row Meta-Llama-3-8B-Instruct; column Specificity; condition Intended attacks |
| Meta-Llama-3-8B-Instruct | Effectiveness | 81% | Safety judgement and risk identification / Intended attacks#S3.T1; row Meta-Llama-3-8B-Instruct; column Effectiveness; condition Intended attacks |
| Meta-Llama-3-8B-Instruct | F1 | 48.32% | Safety judgement and risk identification / Unintended risks#S3.T1; row Meta-Llama-3-8B-Instruct; column F1; condition Unintended risks |
| Meta-Llama-3-8B-Instruct | Recall | 36.00% | Safety judgement and risk identification / Unintended risks#S3.T1; row Meta-Llama-3-8B-Instruct; column Recall; condition Unintended risks |
| Meta-Llama-3-8B-Instruct | Specificity | 76.36% | Safety judgement and risk identification / Unintended risks#S3.T1; row Meta-Llama-3-8B-Instruct; column Specificity; condition Unintended risks |
| Meta-Llama-3-8B-Instruct | Effectiveness | 48% | Safety judgement and risk identification / Unintended risks#S3.T1; row Meta-Llama-3-8B-Instruct; column Effectiveness; condition Unintended risks |
| Llama-2-13b-chat-hf | F1 | 54.80% | Safety judgement and risk identification / All#S3.T1; row Llama-2-13b-chat-hf; column F1; condition All |
| Llama-2-13b-chat-hf | F1 | 60.04% | Safety judgement and risk identification / Intended attacks#S3.T1; row Llama-2-13b-chat-hf; column F1; condition Intended attacks |
| Llama-2-13b-chat-hf | Recall | 80.00% | Safety judgement and risk identification / Intended attacks#S3.T1; row Llama-2-13b-chat-hf; column Recall; condition Intended attacks |
| Llama-2-13b-chat-hf | Specificity | 19.16% | Safety judgement and risk identification / Intended attacks#S3.T1; row Llama-2-13b-chat-hf; column Specificity; condition Intended attacks |
| Llama-2-13b-chat-hf | Effectiveness | 79.5% | Safety judgement and risk identification / Intended attacks#S3.T1; row Llama-2-13b-chat-hf; column Effectiveness; condition Intended attacks |
| Llama-2-13b-chat-hf | F1 | 38.86% | Safety judgement and risk identification / Unintended risks#S3.T1; row Llama-2-13b-chat-hf; column F1; condition Unintended risks |
| Llama-2-13b-chat-hf | Recall | 34.00% | Safety judgement and risk identification / Unintended risks#S3.T1; row Llama-2-13b-chat-hf; column Recall; condition Unintended risks |
| Llama-2-13b-chat-hf | Specificity | 25.45% | Safety judgement and risk identification / Unintended risks#S3.T1; row Llama-2-13b-chat-hf; column Specificity; condition Unintended risks |
| Llama-2-13b-chat-hf | Effectiveness | 38.5% | Safety judgement and risk identification / Unintended risks#S3.T1; row Llama-2-13b-chat-hf; column Effectiveness; condition Unintended risks |
| Llama-2-7b-chat-hf | F1 | 53.74% | Safety judgement and risk identification / All#S3.T1; row Llama-2-7b-chat-hf; column F1; condition All |
| Llama-2-7b-chat-hf | F1 | 62.99% | Safety judgement and risk identification / Intended attacks#S3.T1; row Llama-2-7b-chat-hf; column F1; condition Intended attacks |
| Llama-2-7b-chat-hf | Recall | 91.50% | Safety judgement and risk identification / Intended attacks#S3.T1; row Llama-2-7b-chat-hf; column Recall; condition Intended attacks |
| Llama-2-7b-chat-hf | Specificity | 7.48% | Safety judgement and risk identification / Intended attacks#S3.T1; row Llama-2-7b-chat-hf; column Specificity; condition Intended attacks |
| Llama-2-7b-chat-hf | Effectiveness | 86.75% | Safety judgement and risk identification / Intended attacks#S3.T1; row Llama-2-7b-chat-hf; column Effectiveness; condition Intended attacks |
| Llama-2-7b-chat-hf | F1 | 21.56% | Safety judgement and risk identification / Unintended risks#S3.T1; row Llama-2-7b-chat-hf; column F1; condition Unintended risks |
| Llama-2-7b-chat-hf | Recall | 18.00% | Safety judgement and risk identification / Unintended risks#S3.T1; row Llama-2-7b-chat-hf; column Recall; condition Unintended risks |
| Llama-2-7b-chat-hf | Specificity | 10.91% | Safety judgement and risk identification / Unintended risks#S3.T1; row Llama-2-7b-chat-hf; column Specificity; condition Unintended risks |
| Llama-2-7b-chat-hf | Effectiveness | 17% | Safety judgement and risk identification / Unintended risks#S3.T1; row Llama-2-7b-chat-hf; column Effectiveness; condition Unintended risks |
| Random | F1 | 51.32% | Safety judgement and risk identification / All#S3.T1; row Random; column F1; condition All |
| Random | F1 | 56.34% | Safety judgement and risk identification / Intended attacks#S3.T1; row Random; column F1; condition Intended attacks |
| Random | Recall | 50.00% | Safety judgement and risk identification / Intended attacks#S3.T1; row Random; column Recall; condition Intended attacks |
| Random | Specificity | 50.00% | Safety judgement and risk identification / Intended attacks#S3.T1; row Random; column Specificity; condition Intended attacks |
| Random | Effectiveness | 0% | Safety judgement and risk identification / Intended attacks#S3.T1; row Random; column Effectiveness; condition Intended attacks |
| Random | F1 | 49.14% | Safety judgement and risk identification / Unintended risks#S3.T1; row Random; column F1; condition Unintended risks |
| Random | Recall | 50.00% | Safety judgement and risk identification / Unintended risks#S3.T1; row Random; column Recall; condition Unintended risks |
| Random | Specificity | 50.00% | Safety judgement and risk identification / Unintended risks#S3.T1; row Random; column Specificity; condition Unintended risks |
| Random | Effectiveness | 0% | Safety judgement and risk identification / Unintended risks#S3.T1; row Random; column Effectiveness; condition Unintended risks |
| Vicuna-13b-v1.5 | F1 | 16.93% | Safety judgement and risk identification / All#S3.T1; row Vicuna-13b-v1.5; column F1; condition All |
| Vicuna-13b-v1.5 | F1 | 9.76% | Safety judgement and risk identification / Intended attacks#S3.T1; row Vicuna-13b-v1.5; column F1; condition Intended attacks |
| Vicuna-13b-v1.5 | Recall | 6.00% | Safety judgement and risk identification / Intended attacks#S3.T1; row Vicuna-13b-v1.5; column Recall; condition Intended attacks |
| Vicuna-13b-v1.5 | Specificity | 84.11% | Safety judgement and risk identification / Intended attacks#S3.T1; row Vicuna-13b-v1.5; column Specificity; condition Intended attacks |
| Vicuna-13b-v1.5 | Effectiveness | 10% | Safety judgement and risk identification / Intended attacks#S3.T1; row Vicuna-13b-v1.5; column Effectiveness; condition Intended attacks |
| Vicuna-13b-v1.5 | F1 | 30.30% | Safety judgement and risk identification / Unintended risks#S3.T1; row Vicuna-13b-v1.5; column F1; condition Unintended risks |
| Vicuna-13b-v1.5 | Recall | 20.00% | Safety judgement and risk identification / Unintended risks#S3.T1; row Vicuna-13b-v1.5; column Recall; condition Unintended risks |
| Vicuna-13b-v1.5 | Specificity | 78.18% | Safety judgement and risk identification / Unintended risks#S3.T1; row Vicuna-13b-v1.5; column Specificity; condition Unintended risks |
| Vicuna-13b-v1.5 | Effectiveness | 27% | Safety judgement and risk identification / Unintended risks#S3.T1; row Vicuna-13b-v1.5; column Effectiveness; condition Unintended risks |
| Vicuna-13b-v1.5-16k | F1 | 25.00% | Safety judgement and risk identification / All#S3.T1; row Vicuna-13b-v1.5-16k; column F1; condition All |
| Vicuna-13b-v1.5-16k | F1 | 15.49% | Safety judgement and risk identification / Intended attacks#S3.T1; row Vicuna-13b-v1.5-16k; column F1; condition Intended attacks |
| Vicuna-13b-v1.5-16k | Recall | 11.00% | Safety judgement and risk identification / Intended attacks#S3.T1; row Vicuna-13b-v1.5-16k; column Recall; condition Intended attacks |
| Vicuna-13b-v1.5-16k | Specificity | 71.03% | Safety judgement and risk identification / Intended attacks#S3.T1; row Vicuna-13b-v1.5-16k; column Specificity; condition Intended attacks |
| Vicuna-13b-v1.5-16k | Effectiveness | 18.5% | Safety judgement and risk identification / Intended attacks#S3.T1; row Vicuna-13b-v1.5-16k; column Effectiveness; condition Intended attacks |
| Vicuna-13b-v1.5-16k | F1 | 43.24% | Safety judgement and risk identification / Unintended risks#S3.T1; row Vicuna-13b-v1.5-16k; column F1; condition Unintended risks |
| Vicuna-13b-v1.5-16k | Recall | 32.00% | Safety judgement and risk identification / Unintended risks#S3.T1; row Vicuna-13b-v1.5-16k; column Recall; condition Unintended risks |
| Vicuna-13b-v1.5-16k | Specificity | 70.91% | Safety judgement and risk identification / Unintended risks#S3.T1; row Vicuna-13b-v1.5-16k; column Specificity; condition Unintended risks |
| Vicuna-13b-v1.5-16k | Effectiveness | 37.5% | Safety judgement and risk identification / Unintended risks#S3.T1; row Vicuna-13b-v1.5-16k; column Effectiveness; condition Unintended risks |
| Vicuna-7b-v1.5 | F1 | 18.59% | Safety judgement and risk identification / All#S3.T1; row Vicuna-7b-v1.5; column F1; condition All |
| Vicuna-7b-v1.5 | F1 | 18.25% | Safety judgement and risk identification / Intended attacks#S3.T1; row Vicuna-7b-v1.5; column F1; condition Intended attacks |
| Vicuna-7b-v1.5 | Recall | 12.50% | Safety judgement and risk identification / Intended attacks#S3.T1; row Vicuna-7b-v1.5; column Recall; condition Intended attacks |
| Vicuna-7b-v1.5 | Specificity | 77.10% | Safety judgement and risk identification / Intended attacks#S3.T1; row Vicuna-7b-v1.5; column Specificity; condition Intended attacks |
| Vicuna-7b-v1.5 | Effectiveness | 24.5% | Safety judgement and risk identification / Intended attacks#S3.T1; row Vicuna-7b-v1.5; column Effectiveness; condition Intended attacks |
| Vicuna-7b-v1.5 | F1 | 19.35% | Safety judgement and risk identification / Unintended risks#S3.T1; row Vicuna-7b-v1.5; column F1; condition Unintended risks |
| Vicuna-7b-v1.5 | Recall | 12.00% | Safety judgement and risk identification / Unintended risks#S3.T1; row Vicuna-7b-v1.5; column Recall; condition Unintended risks |
| Vicuna-7b-v1.5 | Specificity | 78.18% | Safety judgement and risk identification / Unintended risks#S3.T1; row Vicuna-7b-v1.5; column Specificity; condition Unintended risks |
| Vicuna-7b-v1.5 | Effectiveness | 25% | Safety judgement and risk identification / Unintended risks#S3.T1; row Vicuna-7b-v1.5; column Effectiveness; condition Unintended risks |
| Vicuna-7b-v1.5-16k | F1 | 29.33% | Safety judgement and risk identification / All#S3.T1; row Vicuna-7b-v1.5-16k; column F1; condition All |
| Vicuna-7b-v1.5-16k | F1 | 25.89% | Safety judgement and risk identification / Intended attacks#S3.T1; row Vicuna-7b-v1.5-16k; column F1; condition Intended attacks |
| Vicuna-7b-v1.5-16k | Recall | 20.00% | Safety judgement and risk identification / Intended attacks#S3.T1; row Vicuna-7b-v1.5-16k; column Recall; condition Intended attacks |
| Vicuna-7b-v1.5-16k | Specificity | 67.76% | Safety judgement and risk identification / Intended attacks#S3.T1; row Vicuna-7b-v1.5-16k; column Specificity; condition Intended attacks |
| Vicuna-7b-v1.5-16k | Effectiveness | 36% | Safety judgement and risk identification / Intended attacks#S3.T1; row Vicuna-7b-v1.5-16k; column Effectiveness; condition Intended attacks |
| Vicuna-7b-v1.5-16k | F1 | 36.88% | Safety judgement and risk identification / Unintended risks#S3.T1; row Vicuna-7b-v1.5-16k; column F1; condition Unintended risks |
| Vicuna-7b-v1.5-16k | Recall | 26.00% | Safety judgement and risk identification / Unintended risks#S3.T1; row Vicuna-7b-v1.5-16k; column Recall; condition Unintended risks |
| Vicuna-7b-v1.5-16k | Specificity | 72.73% | Safety judgement and risk identification / Unintended risks#S3.T1; row Vicuna-7b-v1.5-16k; column Specificity; condition Unintended risks |
| Vicuna-7b-v1.5-16k | Effectiveness | 28.5% | Safety judgement and risk identification / Unintended risks#S3.T1; row Vicuna-7b-v1.5-16k; column Effectiveness; condition Unintended risks |
| Mistral-7B-Instruct-v0.2 | F1 | 27.20% | Safety judgement and risk identification / All#S3.T1; row Mistral-7B-Instruct-v0.2; column F1; condition All |
| Mistral-7B-Instruct-v0.2 | F1 | 24.80% | Safety judgement and risk identification / Intended attacks#S3.T1; row Mistral-7B-Instruct-v0.2; column F1; condition Intended attacks |
| Mistral-7B-Instruct-v0.2 | Recall | 15.50% | Safety judgement and risk identification / Intended attacks#S3.T1; row Mistral-7B-Instruct-v0.2; column Recall; condition Intended attacks |
| Mistral-7B-Instruct-v0.2 | Specificity | 91.12% | Safety judgement and risk identification / Intended attacks#S3.T1; row Mistral-7B-Instruct-v0.2; column Specificity; condition Intended attacks |
| Mistral-7B-Instruct-v0.2 | Effectiveness | 37.5% | Safety judgement and risk identification / Intended attacks#S3.T1; row Mistral-7B-Instruct-v0.2; column Effectiveness; condition Intended attacks |
| Mistral-7B-Instruct-v0.2 | F1 | 32.00% | Safety judgement and risk identification / Unintended risks#S3.T1; row Mistral-7B-Instruct-v0.2; column F1; condition Unintended risks |
| Mistral-7B-Instruct-v0.2 | Recall | 20.00% | Safety judgement and risk identification / Unintended risks#S3.T1; row Mistral-7B-Instruct-v0.2; column Recall; condition Unintended risks |
| Mistral-7B-Instruct-v0.2 | Specificity | 90.91% | Safety judgement and risk identification / Unintended risks#S3.T1; row Mistral-7B-Instruct-v0.2; column Specificity; condition Unintended risks |
| Mistral-7B-Instruct-v0.2 | Effectiveness | 38% | Safety judgement and risk identification / Unintended risks#S3.T1; row Mistral-7B-Instruct-v0.2; column Effectiveness; condition Unintended risks |
| Mistral-7B-Instruct-v0.3 | F1 | 25.65% | Safety judgement and risk identification / All#S3.T1; row Mistral-7B-Instruct-v0.3; column F1; condition All |
| Mistral-7B-Instruct-v0.3 | F1 | 21.99% | Safety judgement and risk identification / Intended attacks#S3.T1; row Mistral-7B-Instruct-v0.3; column F1; condition Intended attacks |
| Mistral-7B-Instruct-v0.3 | Recall | 15.50% | Safety judgement and risk identification / Intended attacks#S3.T1; row Mistral-7B-Instruct-v0.3; column Recall; condition Intended attacks |
| Mistral-7B-Instruct-v0.3 | Specificity | 76.17% | Safety judgement and risk identification / Intended attacks#S3.T1; row Mistral-7B-Instruct-v0.3; column Specificity; condition Intended attacks |
| Mistral-7B-Instruct-v0.3 | Effectiveness | 28% | Safety judgement and risk identification / Intended attacks#S3.T1; row Mistral-7B-Instruct-v0.3; column Effectiveness; condition Intended attacks |
| Mistral-7B-Instruct-v0.3 | F1 | 33.09% | Safety judgement and risk identification / Unintended risks#S3.T1; row Mistral-7B-Instruct-v0.3; column F1; condition Unintended risks |
| Mistral-7B-Instruct-v0.3 | Recall | 23.00% | Safety judgement and risk identification / Unintended risks#S3.T1; row Mistral-7B-Instruct-v0.3; column Recall; condition Unintended risks |
| Mistral-7B-Instruct-v0.3 | Specificity | 70.91% | Safety judgement and risk identification / Unintended risks#S3.T1; row Mistral-7B-Instruct-v0.3; column Specificity; condition Unintended risks |
| Mistral-7B-Instruct-v0.3 | Effectiveness | 38% | Safety judgement and risk identification / Unintended risks#S3.T1; row Mistral-7B-Instruct-v0.3; column Effectiveness; condition Unintended risks |
| Llama-2-7b-chat-hf | F1 | 24.14% | Safety judgement: Llama and Llama Guard comparison / F1#S5.T3; row Llama-2-7b-chat-hf; column F1; condition F1 |
| Llama-2-7b-chat-hf | Recall | 18.67% | Safety judgement: Llama and Llama Guard comparison / Recall#S5.T3; row Llama-2-7b-chat-hf; column Recall; condition Recall |
| Llama-2-7b-chat-hf | Specificity | 59.85% | Safety judgement: Llama and Llama Guard comparison / Specificity#S5.T3; row Llama-2-7b-chat-hf; column Specificity; condition Specificity |
| LlamaGuard-7b | F1 | 0.66% | Safety judgement: Llama and Llama Guard comparison / F1#S5.T3; row LlamaGuard-7b; column F1; condition F1 |
| LlamaGuard-7b | Recall | 0.33% | Safety judgement: Llama and Llama Guard comparison / Recall#S5.T3; row LlamaGuard-7b; column Recall; condition Recall |
| LlamaGuard-7b | Specificity | 100.00% | Safety judgement: Llama and Llama Guard comparison / Specificity#S5.T3; row LlamaGuard-7b; column Specificity; condition Specificity |
| Meta-Llama-3-8B-Instruct | F1 | 55.81% | Safety judgement: Llama and Llama Guard comparison / F1#S5.T3; row Meta-Llama-3-8B-Instruct; column F1; condition F1 |
| Meta-Llama-3-8B-Instruct | Recall | 53.67% | Safety judgement: Llama and Llama Guard comparison / Recall#S5.T3; row Meta-Llama-3-8B-Instruct; column Recall; condition Recall |
| Meta-Llama-3-8B-Instruct | Specificity | 56.88% | Safety judgement: Llama and Llama Guard comparison / Specificity#S5.T3; row Meta-Llama-3-8B-Instruct; column Specificity; condition Specificity |
| Meta-Llama-Guard-2-8B | F1 | 71.84% | Safety judgement: Llama and Llama Guard comparison / F1#S5.T3; row Meta-Llama-Guard-2-8B; column F1; condition F1 |
| Meta-Llama-Guard-2-8B | Recall | 74.00% | Safety judgement: Llama and Llama Guard comparison / Recall#S5.T3; row Meta-Llama-Guard-2-8B; column Recall; condition Recall |
| Meta-Llama-Guard-2-8B | Specificity | 64.31% | Safety judgement: Llama and Llama Guard comparison / Specificity#S5.T3; row Meta-Llama-Guard-2-8B; column Specificity; condition Specificity |
Reading scope & review
Selected original HTML sections and table context; not a complete paper/appendix audit or independent reproduction.
Model-assisted reading; independent evidence review pending.
Check the original source ↗