NUWA Evaluation dossier
ForesightSafety-VLA
A four-quadrant success/safety decomposition avoids hiding unsafe successes.
Evidence update · 2026-10-28
Model-assisted reading · independent evidence and bilingual editorial review pending. Values are source findings or inventory counts, separate from the trend result series.
Source-bound observations (2)
Counts, rates and contrasts have different units. A sample count is not a number of independent experiments. Unknown numerators and interval levels remain unknown.
| System / metric | Reported value | Population and conditions |
|---|---|---|
| Base scenarios | 66 scenarios | Conditions, limitations and sourceInventory count, not outcome denominator or severity. Original source ↗ · Abstract; benchmark design; text lines 00082,00084,00134,00141,00146,00147Original capture 2026-10-28T07:24:44.372873+00:00; sha256:303708eaa7bcf2e1d96cd551279703949de0dff652b1b5b86e0cd2dd5905329a |
| Safety categories | 13 categories | Conditions, limitations and sourceInventory count, not outcome denominator or severity. Original source ↗ · Abstract; benchmark design; text lines 00082,00098,00100,00146Original capture 2026-10-28T07:24:44.372873+00:00; sha256:303708eaa7bcf2e1d96cd551279703949de0dff652b1b5b86e0cd2dd5905329a |
Related reading
NUWA reading connections; these do not imply citation, replication or score equivalence.
Task and measurement
Benchmark protocol profile
- Task / subject
- The protocol covers 13 safety categories, 66 base scenarios and five embodiments. Language and visual variations overlay the base scenarios. Outcomes separate safe success, safe failure, unsafe success and unsafe failure, with cumulative safety cost and exposure measures.
- Measurement target
- How do structural, language and visual perturbations affect embodied safety?
- Interpretation boundary
- Overlay variants are not additional independent base tasks. Results depend on simulation safety thresholds and the evaluated embodiments.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- How do structural, language and visual perturbations affect embodied safety?
- Task population
- The protocol covers 13 safety categories, 66 base scenarios and five embodiments. Language and visual variations overlay the base scenarios. Outcomes separate safe success, safe failure, unsafe success and unsafe failure, with cumulative safety cost and exposure measures.
- Scoring and failure modes
- Retain safe/unsafe success/failure and process-cost components. Language and visual overlays are not extra independent base scenarios.
- System conditions
- Source-specific interfaces and constructed task conditions; this reading does not establish matched comparability with other protocols.
- Statistical meaning
- Retain per-metric population, repeats and interval definitions from the record evidence table. Missing event numerators remain unknown.
- Validity and interpretation
- Overlay variants are not additional independent base tasks. Results depend on simulation safety thresholds and the evaluated embodiments.
- Appropriate research use
- NUWA recommendation: Compare safety components within a shared embodiment and configuration, preserving base-scenario and perturbation identities.
- ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models · §§III–V; discussion; text lines 00082,00084,00134,00141,00146,00147Original resource captured 2026-10-28T07:24:44.372873+00:00; sha256:303708eaa7bcf2e1d96cd551279703949de0dff652b1b5b86e0cd2dd5905329a
Original source charts and materials
Selected original material has a measurement profile. Any numeric transcription remains in the private record evidence table; there is no approved chart or imported figure in this batch.
Related research questions
Autonomous physical harm / Physical action under control →
Editorial scope association; heterogeneous scores are not equivalent.
References
- [1] Mingyang Lyu, Yinqian Sun, et al.. ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models