Risk Evaluation · Benchmark

Detecting Multi-Agent Collusion Through Multi-Agent Interpretability

NUWA research note · independent review pending

First published: 2026-04-01Source checked: 2026-10-25

Original source ↗Cite this record ↓Deception →

NUWA Research Note

Research question Design and assumptions Source findings NUWA interpretation Limits and contrary evidence Research and decision relevance Institutions and roles Original resources

Model-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.

At a glance

NUWA interpretation: high ranking discrimination does not ensure low-false-positive operational detection.

Research question

Can activation-based group detectors generalize from known collusion to new scenarios?

Design and assumptions

Four open-weight models and six probe architectures use Core leave-domain-out tests, then Transfer/Stego tests with five aggregation techniques.

Detecting Multi-Agent Collusion Through Multi-Agent Interpretability — Figure 6 — Figure 6 · Original figure 6. Read alongside the protocol scope and limitations below. Original figure and source ↗

Source findings

Every model reached 1.00 in-distribution AUROC.

On structural Transfer, the best reported method recalled only 0.38 at 1% FPR, despite stronger AUROC results.

NUWA interpretation

NUWA interpretation: high ranking discrimination does not ensure low-false-positive operational detection.

Limits and contrary evidence

Synthetic collusion and small model coverage constrain generalization; appendix hyperparameters were selected using OOD mean AUROC, so tuning visibility matters.

Research and decision relevance

Model recommendation: report strict-threshold recall and separate tuning, training and held-out environments.

Precise source locations

Detecting Multi-Agent Collusion Through Multi-Agent Interpretability · AbstractSelected original resource https://arxiv.org/html/2604.01151v2; sha256:4a8eeea1f124ad8e96c604e36d0ecf2429f2dd528655f3aa19dde2aad705adc7; archived source capture; exact capture time not independently established
Detecting Multi-Agent Collusion Through Multi-Agent Interpretability · §3.2 Activation extraction and probe training: EvaluationSelected original resource https://arxiv.org/html/2604.01151v2; sha256:4a8eeea1f124ad8e96c604e36d0ecf2429f2dd528655f3aa19dde2aad705adc7; archived source capture; exact capture time not independently established
Detecting Multi-Agent Collusion Through Multi-Agent Interpretability · §4.4 Detection at operational thresholdsSelected original resource https://arxiv.org/html/2604.01151v2; sha256:4a8eeea1f124ad8e96c604e36d0ecf2429f2dd528655f3aa19dde2aad705adc7; archived source capture; exact capture time not independently established
Detecting Multi-Agent Collusion Through Multi-Agent Interpretability · Appendix B Logistic Regression Auxiliary ExperimentsSelected original resource https://arxiv.org/html/2604.01151v2; sha256:4a8eeea1f124ad8e96c604e36d0ecf2429f2dd528655f3aa19dde2aad705adc7; archived source capture; exact capture time not independently established

Source checking, extraction, editorial approval and replication are recorded separately. Methodology →

Multi-agent dynamics

NARCBench uses four agents, with two instructed colluders in collusion runs and an all-honest control. Probes learn a deception direction from Core scenarios; five aggregation methods classify whole runs. Evaluation separates held-out Core domains, structurally different Transfer scenarios and a steganographic card-game tier, across four open-weight models.

Institutions and roles

Research by University of Oxford ELLIS Institute Tübingen

Authors (6)

Aaron Rose · Carissa Cullen · Sahar Abdelnabi · Philip Torr · Brandon Gary Kaplowitz · Christian Schroeder de Witt

Evaluation protocols

Detecting Multi-Agent Collusion Through Multi-Agent Interpretability →

Cite and reuse

Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.

BibTeX ↓RIS ↓Record + source metadata · JSON ↓Reading coverage · JSON Suggest a correction

Original publication · BibTeX

@misc{narcbench,
  title = {Detecting Multi-Agent Collusion Through Multi-Agent Interpretability},
  url = {https://arxiv.org/abs/2604.01151v2},
  author = {Aaron Rose and Carissa Cullen and Sahar Abdelnabi and Philip Torr and Brandon Gary Kaplowitz and Christian Schroeder de Witt},
  year = {2026},
  date = {2026-04-01},
  version = {arXiv:2604.01151v2},
  note = {arXiv:2604.01151v2}
}

NUWA record · BibTeX

@misc{narcbench-nuwa,
  title = {Detecting Multi-Agent Collusion Through Multi-Agent Interpretability — NUWA research record},
  url = {https://whitzard.tech/en/nuwa/whitzard-index/work/narcbench/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--narcbench.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--narcbench.json.}
}

Evidence snapshot 2026-10-29.1 · Index updates · RSS

References

[1] Aaron Rose, Carissa Cullen, et al.. Detecting Multi-Agent Collusion Through Multi-Agent Interpretability. 2026-04-01. v2 [arXiv]; Author affiliations on the original paper title page; Abstract; §3.2 Activation extraction and probe training: Evaluation