NUWA Evaluation dossier
Detecting Multi-Agent Collusion Through Multi-Agent Interpretability
Studies detection of covert coordination between agents using observable communication and internal signals.
Task and measurement
Benchmark protocol profile
- Task / subject
- Model or agent system in the original evaluation configuration
- Measurement target
- Studies detection of covert coordination between agents using observable communication and internal signals.
- Interpretation boundary
- Constructed collusion and monitor visibility determine scope; detection does not establish prevalence of spontaneous coordination.
NARCBench uses four agents, with two instructed colluders in collusion runs and an all-honest control. Probes learn a deception direction from Core scenarios; five aggregation methods classify whole runs. Evaluation separates held-out Core domains, structurally different Transfer scenarios and a steganographic card-game tier, across four open-weight models.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can activation-based group detectors generalize from known collusion to new scenarios?
- Task population
- Four open-weight models and six probe architectures use Core leave-domain-out tests, then Transfer/Stego tests with five aggregation techniques.
- Scoring and failure modes
- AUROC plus recall at specified FPR, with group-level aggregation.
- System conditions
- Core leave-domain-out versus zero-shot Transfer/Stego; tuning visibility must be disclosed.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: Every model reached 1.00 in-distribution AUROC. On structural Transfer, the best reported method recalled only 0.38 at 1% FPR, despite stronger AUROC results.
- Validity and interpretation
- Synthetic collusion and small model coverage constrain generalization; appendix hyperparameters were selected using OOD mean AUROC, so tuning visibility matters.
- Appropriate research use
- Model recommendation: report strict-threshold recall and separate tuning, training and held-out environments.
- Detecting Multi-Agent Collusion Through Multi-Agent Interpretability · AbstractSelected original resource https://arxiv.org/html/2604.01151v2; sha256:4a8eeea1f124ad8e96c604e36d0ecf2429f2dd528655f3aa19dde2aad705adc7; archived source capture; exact capture time not independently established
- Detecting Multi-Agent Collusion Through Multi-Agent Interpretability · §3.2 Activation extraction and probe training: EvaluationSelected original resource https://arxiv.org/html/2604.01151v2; sha256:4a8eeea1f124ad8e96c604e36d0ecf2429f2dd528655f3aa19dde2aad705adc7; archived source capture; exact capture time not independently established
- Detecting Multi-Agent Collusion Through Multi-Agent Interpretability · §4.4 Detection at operational thresholdsSelected original resource https://arxiv.org/html/2604.01151v2; sha256:4a8eeea1f124ad8e96c604e36d0ecf2429f2dd528655f3aa19dde2aad705adc7; archived source capture; exact capture time not independently established
- Detecting Multi-Agent Collusion Through Multi-Agent Interpretability · Appendix B Logistic Regression Auxiliary ExperimentsSelected original resource https://arxiv.org/html/2604.01151v2; sha256:4a8eeea1f124ad8e96c604e36d0ecf2429f2dd528655f3aa19dde2aad705adc7; archived source capture; exact capture time not independently established
Original source charts and materials
Related research questions
References
- [1] Aaron Rose, Carissa Cullen, et al.. Detecting Multi-Agent Collusion Through Multi-Agent Interpretability. 2026-04-01. v2 [arXiv]; Author affiliations on the original paper title page; Figure 6