WhitzardNUWA Lab ↗
NUWANUWA INDEX

Search NUWA Index

Research · Benchmarks · Findings · Perspectives

中文
← Library

Risk Evaluation · Dataset

MALT: A Dataset of Natural and Prompted Behaviors That Threaten Eval Integrity

NUWA research note · independent review pending

First published: Source checked: 2026-10-25
Risk monitoring
Original source ↗Cite this record ↓Deception →
NUWA Research NoteResearch questionDesign and assumptionsSource findingsNUWA interpretationLimits and contrary evidenceResearch and decision relevanceInstitutions and rolesOriginal resources

Model-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.

At a glance

NUWA interpretation: a labeled detection dataset should not be mistaken for an unbiased prevalence sample.

Research question

Can manually labeled agent transcripts ground checks for evaluation-integrity failures?

Design and assumptions

MALT collects 10,919 transcripts from 403 tasks and 21 models, mixing natural and prompted behavior; public and reviewed subsets differ.

Source findings

2,690 transcripts were manually reviewed: 1,014 benign and 1,676 with observed behavior.

The remaining 8,229 were not thoroughly checked; monitoring results use the fully reviewed subset.

NUWA interpretation

NUWA interpretation: a labeled detection dataset should not be mistaken for an unbiased prevalence sample.

Limits and contrary evidence

Collection includes anomaly searches and prompted incidents; estimated label error in the unchecked subset was 6%, and public release is task-filtered.

Research and decision relevance

Model recommendation: keep reviewed, public and prompted/unprompted populations distinct in monitoring claims.

Precise source locations

Source checking, extraction, editorial approval and replication are recorded separately. Methodology →

Institutions and roles

Published by METR

Cite and reuse

Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.

BibTeX ↓RIS ↓Record + source metadata · JSON ↓Reading coverage · JSONSuggest a correction
Original publication · BibTeX
@misc{malt,
  title = {MALT: A Dataset of Natural and Prompted Behaviors That Threaten Eval Integrity},
  url = {https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/},
  year = {2025},
  date = {2025-10-14},
  version = {Source capture 2026-10-25},
  note = {Source capture 2026-10-25}
}
NUWA record · BibTeX
@misc{malt-nuwa,
  title = {MALT: A Dataset of Natural and Prompted Behaviors That Threaten Eval Integrity — NUWA research record},
  url = {https://whitzard.tech/en/nuwa/whitzard-index/work/malt/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--malt.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--malt.json.}
}

Evidence snapshot 2026-10-29.1 · Index updates · RSS

References

  1. [1] MALT: A Dataset of Natural and Prompted Behaviors That Threaten Eval Integrity. 2025-10-14 ; Official publication masthead and article byline; Summary; Dataset details