WhitzardNUWA Lab ↗
NUWANUWA INDEX

Search NUWA Index

Research · Benchmarks · Findings · Perspectives

中文
← Library

Institution

University of California, Berkeley

Roles and affiliations follow the cited source. A source check is not an independent replication.

More filters →

University of California, Berkeley ↗

The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark

QuestionCan agents recover semantics from realistic binaries without source access?

Source summaryGPT-5.6 Sol fully solved 80/254 graded instances, versus 60/223 for Claude Fable 5.1.

NUWA research note · independent review pending

CybersecurityCapability

ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?

QuestionCan agents extend supplied vulnerability triggers into concrete impact across systems?

Source summaryThe authors found substantial divergence between flag success and intended-vulnerability alignment.

NUWA research note · independent review pending

CybersecurityCapability

Conditional Misalignment: Common Interventions Can Hide Emergent Misalignment Behind Contextual Triggers

QuestionDo apparent mitigations remove broad misalignment, or make its expression conditional on training-like cues?

Source summaryStandard questions can look aligned while training-like cues elicit broader failures.

NUWA research note · independent review pending

Risk EvaluationTechnical report

Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

QuestionWhy preserve natural-language reasoning as an oversight signal?

Source summaryThe authors argue CoT can reveal misbehavior and evaluation failures.

NUWA research note · independent review pending · Contains: question / method / findings / interpretation / limits

Risk monitoringCross-cutting research

CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale

QuestionCan agents reproduce real historical software vulnerabilities at scale, and how do input information, model and agent configuration change that ability?

Source summaryThe difficulty ladder demonstrates that richer diagnostic information makes reproduction easier for the examined configuration (§4, Figure 6).

NUWA research note · independent review pending · Contains: question / method / limits

CybersecurityCapability

BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems

QuestionCan one framework distinguish detection, exploitation and patching across real applications?

Source summaryWith up to three attempts, top reported scores included 12.5% Detect, 67.5% Exploit and 90% Patch from different agent setups.

NUWA research note · independent review pending

CybersecurityCapability

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

QuestionDoes a narrow training objective generalize into unrelated misaligned responses rather than merely accepting harmful requests?

Source summaryEducational framing changes generalization despite matching insecure code.

NUWA research note · independent review pending · Contains: question / method / findings / interpretation / limits

JudgeBench: A Benchmark for Evaluating LLM-based Judges

QuestionCan LLM judges distinguish objectively correct from incorrect sophisticated responses?

Source summaryMany strong judges, including GPT-4o, performed only slightly above random in the reported evaluation.

NUWA research note · independent review pending · Contains: question / method / findings / interpretation / limits

Cross-cutting researchCapability

The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning

QuestionCan a public proxy benchmark track hazardous knowledge while supporting mitigation research?

Source summaryThe paper evaluates knowledge-reduction methods alongside general-capability retention.

NUWA research note · independent review pending · Contains: question / method / limits

CBRNCapability

HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal

QuestionCan automated red-team methods and defenses be compared with consistent behavior-level scoring?

Source summaryContextual behaviors had higher attack success than some other functional categories.

NUWA research note · independent review pending · Contains: question / method / limits

Cross-cutting researchBehavior

Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark

QuestionHow does reward optimization trade off with ethical conduct in social decision games?

Source summaryReward optimization created some tension with ethical conduct.

NUWA research note · independent review pending

DeceptionCapability