NUWA Evaluation dossier
LinuxArena
Can software-agent safeguards be assessed in richer operational environments?
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
LinuxArena · Main task success rate
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
12 observations · 0 dated
12 observations lack a verified model release date. They remain in Data and Systems.
Select a point or table row to inspect its conditions and provenance.
12 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
 Claude Opus 4.6 | 66%Main task success rate | Incomplete | Main task performance; figure reports n=400 each; task count versus repeated runs not separately disclosedLinuxArena | Not disclosedIntroducing LinuxArena | Release —Report 2026-04-20 |
| Claude Sonnet 4.6 | 64%Main task success rate | Incomplete | Main task performance; figure reports n=400 each; task count versus repeated runs not separately disclosedLinuxArena | Not disclosedIntroducing LinuxArena | Release —Report 2026-04-20 |
 Gemini 3.1 Pro | 60%Main task success rate | Incomplete | Main task performance; figure reports n=400 each; task count versus repeated runs not separately disclosedLinuxArena | Not disclosedIntroducing LinuxArena | Release —Report 2026-04-20 |
| Claude Code (Opus 4.6) | 59%Main task success rate | Incomplete | Main task performance; figure reports n=400 each; task count versus repeated runs not separately disclosedLinuxArena | Not disclosedIntroducing LinuxArena | Release —Report 2026-04-20 |
 GLM 5 | 55%Main task success rate | Incomplete | Main task performance; figure reports n=400 each; task count versus repeated runs not separately disclosedLinuxArena | Not disclosedIntroducing LinuxArena | Release —Report 2026-04-20 |
| Codex (GPT-5.4) | 53%Main task success rate | Incomplete | Main task performance; figure reports n=400 each; task count versus repeated runs not separately disclosedLinuxArena | Not disclosedIntroducing LinuxArena | Release —Report 2026-04-20 |
 GPT-5.4 | 52%Main task success rate | Incomplete | Main task performance; figure reports n=400 each; task count versus repeated runs not separately disclosedLinuxArena | Not disclosedIntroducing LinuxArena | Release —Report 2026-04-20 |
| Gemini 3 Flash | 52%Main task success rate | Incomplete | Main task performance; figure reports n=400 each; task count versus repeated runs not separately disclosedLinuxArena | Not disclosedIntroducing LinuxArena | Release —Report 2026-04-20 |
| Claude Haiku 4.5 | 51%Main task success rate | Incomplete | Main task performance; figure reports n=400 each; task count versus repeated runs not separately disclosedLinuxArena | Not disclosedIntroducing LinuxArena | Release —Report 2026-04-20 |
| GPT-5 Mini | 41%Main task success rate | Incomplete | Main task performance; figure reports n=400 each; task count versus repeated runs not separately disclosedLinuxArena | Not disclosedIntroducing LinuxArena | Release —Report 2026-04-20 |
| GPT-5 Nano | 24%Main task success rate | Incomplete | Main task performance; figure reports n=400 each; task count versus repeated runs not separately disclosedLinuxArena | Not disclosedIntroducing LinuxArena | Release —Report 2026-04-20 |
 Grok 4.1 | 20%Main task success rate | Incomplete | Main task performance; figure reports n=400 each; task count versus repeated runs not separately disclosedLinuxArena | Not disclosedIntroducing LinuxArena | Release —Report 2026-04-20 |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- Pairs legitimate engineering tasks with scored side tasks in running software systems.
- Measurement target
- Can software-agent safeguards be assessed in richer operational environments?
- Scoring / outcome
- Main task success rate · Main task success rate
- Interpretation boundary
- Security-critical task selection and elicitation affect transfer; article figures were not numerically digitized.
A reported evaluation
The source evaluates Claude Opus 4.6 · Main task performance; figure reports n=400 each; task count versus repeated runs not separately disclosed.
Run configuration: LinuxArena.
Reported measurement: Main task success rate · 66%.
Main task success rate 400
Separates sabotage capability, monitor detection and control-protocol usefulness. Human-crafted trajectories expose weaknesses beyond ordinary agent-generated attacks. Article image labels report Opus4.6 main-task success66% (n=400), while its undetected sabotage is25% under a GPT-5 Nano monitor at1%step FPR. Strategy assistance raises that sabotage figure to49%, and human-crafted LaStraj trajectories reach94%. The same monitor’s safety is75% against Opus attacks,51% against strategy-aided Opus attacks and5% against LaStraj. These are distinct metrics with different denominators.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Sabotage, monitor detection and deployment control in software-engineering environments.
- Task population
- 20 software environments pair legitimate engineering tasks with scored security side tasks, including running services and databases.
- Scoring and failure modes
- Sabotage means causing harm without being caught; safety means detection before harm. Usefulness is the proportion of main tasks solved.
- System conditions
- Article evaluates Opus4.6 attacks against GPT-5 Nano monitoring at1% action-level FPR, with separate baseline, strategy-augmented and human-crafted attacks.
- Statistical meaning
- Not established by the retained material
- Validity and interpretation
- Tasks deliberately place agents near security-critical resources. All28explicit labels in article images1–3 are transcribed; graphical interval type/endpoints and exact sabotage denominators are not disclosed. Pareto frontier coordinates are not digitized.
- Appropriate research use
- Not established by the retained material
- Introducing LinuxArena · L1-L36; L37-L570222314816805662a18563e24991608fb94644e3cd2919616b6c84d5f683b076
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
Introducing LinuxArena
Introducing LinuxArena
Actual evaluators: Not disclosed
28 result records · 14 reported system names
Protocols and comparison decisions (0)
Original source charts and materials
Security-critical task selection and elicitation affect transfer; article figures were not numerically digitized.
Related research questions
References
Numerical results are reported by the cited authors.
- [1] Introducing LinuxArena. 2026-04-20 ; Article image 1; label Claude Opus 4.6; Main task performance; figure reports n=400 each; task count versus repeated runs not separately disclosed