{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"narcbench","title":"Detecting Multi-Agent Collusion Through Multi-Agent Interpretability","aliases":[],"authors":["Aaron Rose","Carissa Cullen","Sahar Abdelnabi","Philip Torr","Brandon Gary Kaplowitz","Christian Schroeder de Witt"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"narcbench-publication","attributionSourceId":"narcbench-publication-author-source","originalUrl":"https://arxiv.org/abs/2604.01151v2","publicationDate":"2026-04-01","updatedOn":"2026-10-25","areas":["deception"],"scopeNote":{"en":"Relevance is assigned to the research question and stated conditions of the work. A topic tag is not a risk rating.","zh":"风险关联依据研究问题及明确条件建立，主题标签不代表风险评级。"},"protocolIds":["narcbench"],"contributions":[{"organizationId":"oxford","role":"research","sourceId":"narcbench-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"ellis","role":"research","sourceId":"narcbench-publication-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Studies detection of covert coordination between agents using observable communication and internal signals.","zh":"研究利用可观察通信及内部信号，检测智能体间的隐蔽协调。"},"note":{"question":{"en":"Can aggregating internal signals across agents detect instructed collusion when the interaction environment changes?","zh":"交互环境变化后，汇总多个智能体的内部信号能否识别被明确指派的串谋？"},"method":{"en":"NARCBench uses four agents, with two instructed colluders in collusion runs and an all-honest control. Probes learn a deception direction from Core scenarios; five aggregation methods classify whole runs. Evaluation separates held-out Core domains, structurally different Transfer scenarios and a steganographic card-game tier, across four open-weight models.","zh":"NARCBench 使用四个智能体：串谋实验明确指定两个串谋者，对照实验全部诚实。探针从 Core 场景学习欺骗方向，再用五种汇总方法对整次运行分类。四个开放权重模型分别在留出 Core 领域、结构不同的 Transfer 场景和隐写纸牌游戏层级接受评估。"},"finding":{"en":"The reported activation-based detectors separate collusion from honest controls in the tested settings, including transfer settings. Transfer is harder than in-distribution detection, and no aggregation method is best across all models and tiers.","zh":"报告中的激活检测器能在所测条件下区分串谋与诚实对照，包括迁移场景。迁移检测比同分布检测更困难，且没有一种汇总方法在所有模型和层级上均最佳。"},"supports":{"en":"The study supports group-level white-box signals as a possible complement to transcript monitoring. Its detection results concern deliberately constructed and audited collusion runs.","zh":"研究支持将群体层面的白盒信号作为交互记录监测的潜在补充；检测结果针对刻意构造并经过核查的串谋运行。"},"limits":{"en":"Robustness against agents trying to evade activation monitoring is untested. Probes may exploit spurious correlations; labels exclude runs that fail strict coordination/execution criteria. Four-agent groups with two colluders do not establish performance for arbitrary coalition sizes or large deployments.","zh":"尚未测试智能体主动规避激活监测时的稳健性。探针可能利用伪相关；标签排除了不满足严格协作与执行标准的运行。四个智能体、两个串谋者的实验不能确定任意联盟规模或大型部署中的表现。"},"connections":null},"resources":[],"noteStatus":{"question":"verified","method":"verified","finding":"verified","supports":"verified","limits":"verified","connections":"limited"},"noteAvailability":{"en":"The substantive sections shown here were checked against the original source. Unfilled sections are omitted; source checking is not experimental replication.","zh":"此处展示的实质性章节已对照原始来源核查；未完成的章节已省略。来源核查不等于实验复现。"},"noteSourceLocator":"arXiv:2604.01151v2, §§3.1–3.4, 4.1–4.6; Appendices F.7, G.9, H.8","tags":["risk-monitoring"]},"reading":{"id":"narcbench","title":"Detecting Multi-Agent Collusion Through Multi-Agent Interpretability","en":{"question":"Can activation-based group detectors generalize from known collusion to new scenarios?","design":"Four open-weight models and six probe architectures use Core leave-domain-out tests, then Transfer/Stego tests with five aggregation techniques.","findings":["Every model reached 1.00 in-distribution AUROC.","On structural Transfer, the best reported method recalled only 0.38 at 1% FPR, despite stronger AUROC results."],"interpretation":"NUWA interpretation: high ranking discrimination does not ensure low-false-positive operational detection.","limits":"Synthetic collusion and small model coverage constrain generalization; appendix hyperparameters were selected using OOD mean AUROC, so tuning visibility matters.","decisionRelevance":"Model recommendation: report strict-threshold recall and separate tuning, training and held-out environments.","takeaway":"NUWA interpretation: high ranking discrimination does not ensure low-false-positive operational detection.","connections":[],"scoring":"AUROC plus recall at specified FPR, with group-level aggregation.","conditions":"Core leave-domain-out versus zero-shot Transfer/Stego; tuning visibility must be disclosed.","statistics":"Every model reached 1.00 in-distribution AUROC. On structural Transfer, the best reported method recalled only 0.38 at 1% FPR, despite stronger AUROC results.","usage":"Model recommendation: report strict-threshold recall and separate tuning, training and held-out environments."},"zh":{"question":"激活群体检测能否从已知合谋泛化到新场景？","design":"四个开放模型、六种探针架构，在Core留域验证，再以五种聚合方法测Transfer与Stego。","findings":["各模型分布内AUROC均1.00。","结构迁移上最佳方法在1%误报时召回仅0.38，尽管AUROC较高。"],"interpretation":"NUWA 解读：排序判别高不保证低误报部署检出。","limits":"合成合谋与少量模型限制泛化，附录使用OOD均值选超参数，须明确调参可见性。","decisionRelevance":"模型建议：报告严格阈值召回，并分开调参、训练与保留环境。","takeaway":"NUWA 解读：排序判别高不保证低误报部署检出。","connections":[],"scoring":"群体聚合AUROC及指定误报率下召回。","conditions":"Core留域与Transfer/Stego迁移分开，披露调参可见性。","statistics":"各模型分布内AUROC均1.00。 结构迁移上最佳方法在1%误报时召回仅0.38，尽管AUROC较高。","usage":"模型建议：报告严格阈值召回，并分开调参、训练与保留环境。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"narcbench-publication-author-source","url":"https://arxiv.org/html/2604.01151v2","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2604.01151v2; sha256:4a8eeea1f124ad8e96c604e36d0ecf2429f2dd528655f3aa19dde2aad705adc7; archived source capture; exact capture time not independently established","artifactSha256":"4a8eeea1f124ad8e96c604e36d0ecf2429f2dd528655f3aa19dde2aad705adc7"},{"sourceId":"narcbench-publication-author-source","url":"https://arxiv.org/html/2604.01151v2","locator":"§3.2 Activation extraction and probe training: Evaluation","version":"Selected original resource https://arxiv.org/html/2604.01151v2; sha256:4a8eeea1f124ad8e96c604e36d0ecf2429f2dd528655f3aa19dde2aad705adc7; archived source capture; exact capture time not independently established","artifactSha256":"4a8eeea1f124ad8e96c604e36d0ecf2429f2dd528655f3aa19dde2aad705adc7"},{"sourceId":"narcbench-publication-author-source","url":"https://arxiv.org/html/2604.01151v2","locator":"§4.4 Detection at operational thresholds","version":"Selected original resource https://arxiv.org/html/2604.01151v2; sha256:4a8eeea1f124ad8e96c604e36d0ecf2429f2dd528655f3aa19dde2aad705adc7; archived source capture; exact capture time not independently established","artifactSha256":"4a8eeea1f124ad8e96c604e36d0ecf2429f2dd528655f3aa19dde2aad705adc7"},{"sourceId":"narcbench-publication-author-source","url":"https://arxiv.org/html/2604.01151v2","locator":"Appendix B Logistic Regression Auxiliary Experiments","version":"Selected original resource https://arxiv.org/html/2604.01151v2; sha256:4a8eeea1f124ad8e96c604e36d0ecf2429f2dd528655f3aa19dde2aad705adc7; archived source capture; exact capture time not independently established","artifactSha256":"4a8eeea1f124ad8e96c604e36d0ecf2429f2dd528655f3aa19dde2aad705adc7"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"d56e0ea9a73cf7f9bce84289c6447d9cf298ba881d963b2482616f9ebca1224c"}},"sources":[{"id":"narcbench-publication","title":"Detecting Multi-Agent Collusion Through Multi-Agent Interpretability","url":"https://arxiv.org/abs/2604.01151v2","version":"arXiv:2604.01151v2","publicationDate":"2026-04-01","datePrecision":"day","type":"paper","verifiedOn":"2026-10-25","documentId":"arxiv:2604.01151","authors":["Aaron Rose","Carissa Cullen","Sahar Abdelnabi","Philip Torr","Brandon Gary Kaplowitz","Christian Schroeder de Witt"],"aliases":["https://doi.org/10.48550/arXiv.2604.01151","https://arxiv.org/abs/2604.01151","https://arxiv.org/pdf/2604.01151"]}]}