NUWA Evaluation dossier
ASIMOV v2
Perception of physical danger and safety interventions.
Evidence update · 2026-10-28
Model-assisted reading · independent evidence and bilingual editorial review pending. Values are source findings or inventory counts, separate from the trend result series.
Hazard recognition, chat refusal, control actions and damage are distinct endpoints. RoboHarm shows interface-sensitive refusal; the simulation benchmarks add unsafe success, damage and process costs while still needing real-environment calibration.
Related reading
NUWA reading connections; these do not imply citation, replication or score equivalence.
Sources for this interpretation
- Refuses in Chat, Complies with Arms: LLM Refusal Does Not Survive Robot Control · Abstract; §§2–4; discussion; text lines 00044,00055,00061Original resource captured 2026-10-28T07:24:43.704880+00:00; sha256:c5b843fd1a59541e7c7713423f4c765813de1b6dc8cb85bc7aa5913c0c945572
- OopsieVerse: A Safety Benchmark with Damage-Aware Simulation for Robot Manipulation · §§III–V; limitations; text lines 00029,00303,00307,00309Original resource captured 2026-10-28T07:24:43.858466+00:00; sha256:e5c0c24c6eb494b5bef1612f1f6d5381aa9972a4aaa8937de7dcdb3e046a48d0
- ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models · §§III–V; discussion; text lines 00082,00084,00134,00141,00146,00147Original resource captured 2026-10-28T07:24:44.372873+00:00; sha256:303708eaa7bcf2e1d96cd551279703949de0dff652b1b5b86e0cd2dd5905329a
Task and measurement
Benchmark protocol profile
- Task / subject
- Model interpreting multimodal physical-risk scenes
- Measurement target
- Multimodal safety reasoning on generated scenes.
- Interpretation boundary
- Scene-based reasoning does not certify a deployed robot's physical safety.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can embodied models recognize injury risks and obey embodiment-specific safety constraints?
- Task population
- ASIMOV-2.0 derives text scenarios from injury narratives and constraints, then generates image/video variants; recognition and intervention are separate tasks.
- Scoring and failure modes
- Separate accuracy measures for latent-risk recognition and high-severity judgments; intervention is a different task.
- System conditions
- Text, image and video conditions and embodiment-specific constraints must remain separate.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: Three leading models averaged 92.3% text latent-risk accuracy and 88.7% high-severity accuracy in the reported comparison. Smaller variants had material gaps from their corresponding larger models (§3).
- Validity and interpretation
- Generative stimuli and scenario judgments do not establish closed-loop robot safety; the selected reading does not audit synthetic-image fidelity.
- Appropriate research use
- Model recommendation: retain modality, model variant and intervention task when using these results to select embodied-system tests.
- Can AI Perceive Physical Danger and Intervene? Thanks: {abjindal, dklakashnikov, oscarchang, divyagari, animajumdar, sindhwani}@google.com · §2 ASIMOV-2.0: A New Physical Safety BenchmarkSelected original resource https://arxiv.org/html/2509.21651; sha256:e9cdfcb13b688a5152a4c9475996663228d49e644405112f6073ceb9934e283d; archived source capture; exact capture time not independently established
- Can AI Perceive Physical Danger and Intervene? Thanks: {abjindal, dklakashnikov, oscarchang, divyagari, animajumdar, sindhwani}@google.com · §3 Evaluations and InsightsSelected original resource https://arxiv.org/html/2509.21651; sha256:e9cdfcb13b688a5152a4c9475996663228d49e644405112f6073ceb9934e283d; archived source capture; exact capture time not independently established
Original source charts and materials
Related research questions
Autonomous physical harm / Physical hazard recognition →
The source directly measures the stated construct under its protocol.
References
- [1] Abhishek Jindal, Dmitry Kalashnikov, et al.. Can AI Perceive Physical Danger and Intervene?. ASIMOV v2 / arXiv:2509.21651 ; Project page author affiliation list
- [2] Abhishek Jindal, Dmitry Kalashnikov, et al.. Can AI Perceive Physical Danger and Intervene? ; Figure 7