Risk Evaluation · Benchmark

HCAST: Human-Calibrated Autonomy Software Tasks

NUWA research note · independent review pending

First published: 2025-03-21Source checked: 2026-10-25

Original source ↗Cite this record ↓Autonomy →

NUWA Research Note

Research question Design and assumptions Source findings NUWA interpretation Limits and contrary evidence Research and decision relevance Institutions and roles Original resources

Model-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.

At a glance

NUWA interpretation: task-time calibration helps contextualize success, but observed and forecast durations are different evidence.

Research question

Can human task duration ground autonomy capability measurement?

Design and assumptions

189 software, ML, cyber and reasoning tasks include 563 human attempts totaling over 1,500 hours, with manual task quality assurance.

HCAST — Figure 6 — Figure 6 · Original figure 6. Read alongside the protocol scope and limitations below. Original figure and source ↗

Source findings

Agents succeeded 70–80% on sub-hour tasks and below 20% on tasks estimated above four hours.

Some task-time estimates rely on forecasts rather than successful human attempts (§3.2).

NUWA interpretation

NUWA interpretation: task-time calibration helps contextualize success, but observed and forecast durations are different evidence.

Limits and contrary evidence

Long-task human baselines face selection and logistical biases; the tasks incompletely represent real work (§5).

Research and decision relevance

Model recommendation: retain time-estimate source and task family when interpreting autonomy horizons.

Precise source locations

HCAST: Human-Calibrated Autonomy Software Tasks · AbstractSelected original resource https://arxiv.org/html/2503.17354; sha256:74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8; archived source capture; exact capture time not independently established
HCAST: Human-Calibrated Autonomy Software Tasks · §2 Benchmark Creation ProcessSelected original resource https://arxiv.org/html/2503.17354; sha256:74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8; archived source capture; exact capture time not independently established
HCAST: Human-Calibrated Autonomy Software Tasks · §3.2 Human Time EstimatesSelected original resource https://arxiv.org/html/2503.17354; sha256:74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8; archived source capture; exact capture time not independently established
HCAST: Human-Calibrated Autonomy Software Tasks · §5 DiscussionSelected original resource https://arxiv.org/html/2503.17354; sha256:74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8; archived source capture; exact capture time not independently established

Source checking, extraction, editorial approval and replication are recorded separately. Methodology →

Institutions and roles

Research by METR Equistamp

Authors (22)

David Rein · Joel Becker · Amy Deng · Seraphina Nix · Chris Canal · Daniel O'Connel · Pip Arnott · Ryan Bloom · Thomas Broadley · Katharyn Garcia · Brian Goodrich · Max Hasin · Sami Jawhar · Megan Kinniment · Thomas Kwa · Aron Lajko · Nate Rush · Lucas Jun Koba Sato · Sydney Von Arx · Ben West · Lawrence Chan · Elizabeth Barnes

Evaluation protocols

HCAST →

Cite and reuse

Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.

BibTeX ↓RIS ↓Record + source metadata · JSON ↓Reading coverage · JSON Suggest a correction

Original publication · BibTeX

@misc{hcast,
  title = {HCAST: Human-Calibrated Autonomy Software Tasks},
  url = {https://arxiv.org/abs/2503.17354},
  author = {David Rein and Joel Becker and Amy Deng and Seraphina Nix and Chris Canal and Daniel O'Connel and Pip Arnott and Ryan Bloom and Thomas Broadley and Katharyn Garcia and Brian Goodrich and Max Hasin and Sami Jawhar and Megan Kinniment and Thomas Kwa and Aron Lajko and Nate Rush and Lucas Jun Koba Sato and Sydney Von Arx and Ben West and Lawrence Chan and Elizabeth Barnes},
  year = {2025},
  date = {2025-03-21},
  version = {arXiv:2503.17354v1},
  note = {arXiv:2503.17354v1}
}

NUWA record · BibTeX

@misc{hcast-nuwa,
  title = {HCAST: Human-Calibrated Autonomy Software Tasks — NUWA research record},
  url = {https://whitzard.tech/en/nuwa/whitzard-index/work/hcast/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--hcast.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--hcast.json.}
}

Evidence snapshot 2026-10-29.1 · Index updates · RSS

References

[1] David Rein, Joel Becker, et al.. HCAST: Human-Calibrated Autonomy Software Tasks. 2025-03-21. v1 [arXiv]; Author affiliations on the original paper title page; Abstract; §2 Benchmark Creation Process