NUWA Evaluation dossier
RoboHarm
Text refusal and embodied action safety measure different failure paths.
Evidence update · 2026-10-28
Model-assisted reading · independent evidence and bilingual editorial review pending. Values are source findings or inventory counts, separate from the trend result series.
Source-bound observations (2)
Counts, rates and contrasts have different units. A sample count is not a number of independent experiments. Unknown numerators and interval levels remain unknown.
| System / metric | Reported value | Population and conditions |
|---|---|---|
| Refusal rate | 18 % | Denominator: 612 trials Conditions, limitations and source{
"interface": "real-robot end-effector"
}No exact refusal numerator inferred; interface/model populations differ. Refusal is not executed harm. Original source ↗ · Abstract; real-robot evaluation; text lines 00044,00055,00061Original capture 2026-10-28T07:24:43.704880+00:00; sha256:c5b843fd1a59541e7c7713423f4c765813de1b6dc8cb85bc7aa5913c0c945572 |
| Refusal rate | 41 % | Denominator: 252 trials Conditions, limitations and source{
"interface": "real-robot joint"
}No exact refusal numerator inferred; interface/model populations differ. Refusal is not executed harm. Original source ↗ · Abstract; real-robot evaluation; text lines 00044,00055,00252Original capture 2026-10-28T07:24:43.704880+00:00; sha256:c5b843fd1a59541e7c7713423f4c765813de1b6dc8cb85bc7aa5913c0c945572 |
Related reading
NUWA reading connections; these do not imply citation, replication or score equivalence.
Task and measurement
Benchmark protocol profile
- Task / subject
- The study compares chat with robot-control interfaces and separates real-robot end-effector and joint-control settings. The real-robot conditions involve different interface/model populations, with separate trial counts.
- Measurement target
- Does a refusal observed in chat persist when the model controls a robot?
- Interpretation boundary
- The settings are not a matched model-only comparison. Rounded rates do not identify exact refusal numerators, and refusal is not a physical harm measurement.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Does a refusal observed in chat persist when the model controls a robot?
- Task population
- The study compares chat with robot-control interfaces and separates real-robot end-effector and joint-control settings. The real-robot conditions involve different interface/model populations, with separate trial counts.
- Scoring and failure modes
- Refusal is the measured endpoint. The 18% and 41% real-robot rates use different interfaces and 612/252 trials, and do not measure executed harm.
- System conditions
- Source-specific interfaces and constructed task conditions; this reading does not establish matched comparability with other protocols.
- Statistical meaning
- Retain per-metric population, repeats and interval definitions from the record evidence table. Missing event numerators remain unknown.
- Validity and interpretation
- The settings are not a matched model-only comparison. Rounded rates do not identify exact refusal numerators, and refusal is not a physical harm measurement.
- Appropriate research use
- NUWA recommendation: Pair hazard recognition with action-level evaluation; retain the interface, embodiment and outcome definition.
- Refuses in Chat, Complies with Arms: LLM Refusal Does Not Survive Robot Control · Abstract; §§2–4; discussion; text lines 00044,00055,00061Original resource captured 2026-10-28T07:24:43.704880+00:00; sha256:c5b843fd1a59541e7c7713423f4c765813de1b6dc8cb85bc7aa5913c0c945572
Original source charts and materials
Selected original material has a measurement profile. Any numeric transcription remains in the private record evidence table; there is no approved chart or imported figure in this batch.
Related research questions
Autonomous physical harm / Physical action under control →
Editorial scope association; heterogeneous scores are not equivalent.
References
- [1] Edward Sun, Christopher Leet, et al.. Refuses in Chat, Complies with Arms: LLM Refusal Does Not Survive Robot Control. 2026-10-03