NUWA Evaluation dossier
HCAST
Uses human-calibrated software tasks to relate agent task completion to measured human effort.
Task and measurement
Benchmark protocol profile
- Task / subject
- Model or agent system under the original task configuration
- Measurement target
- Uses human-calibrated software tasks to relate agent task completion to measured human effort.
- Interpretation boundary
- Human effort calibration depends on the participant population, task distribution and agent configuration.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can human task duration ground autonomy capability measurement?
- Task population
- 189 software, ML, cyber and reasoning tasks include 563 human attempts totaling over 1,500 hours, with manual task quality assurance.
- Scoring and failure modes
- Task success related to observed or forecast human time-to-complete.
- System conditions
- 189 tasks and 563 human attempts; forecast-based times more common for longer tasks.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: Agents succeeded 70–80% on sub-hour tasks and below 20% on tasks estimated above four hours. Some task-time estimates rely on forecasts rather than successful human attempts (§3.2).
- Validity and interpretation
- Long-task human baselines face selection and logistical biases; the tasks incompletely represent real work (§5).
- Appropriate research use
- Model recommendation: retain time-estimate source and task family when interpreting autonomy horizons.
- HCAST: Human-Calibrated Autonomy Software Tasks · AbstractSelected original resource https://arxiv.org/html/2503.17354; sha256:74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8; archived source capture; exact capture time not independently established
- HCAST: Human-Calibrated Autonomy Software Tasks · §2 Benchmark Creation ProcessSelected original resource https://arxiv.org/html/2503.17354; sha256:74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8; archived source capture; exact capture time not independently established
- HCAST: Human-Calibrated Autonomy Software Tasks · §3.2 Human Time EstimatesSelected original resource https://arxiv.org/html/2503.17354; sha256:74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8; archived source capture; exact capture time not independently established
- HCAST: Human-Calibrated Autonomy Software Tasks · §5 DiscussionSelected original resource https://arxiv.org/html/2503.17354; sha256:74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8; archived source capture; exact capture time not independently established
Original source charts and materials
Related research questions
References
- [1] David Rein, Joel Becker, et al.. HCAST: Human-Calibrated Autonomy Software Tasks. 2025-03-21. v1 [arXiv]; Author affiliations on the original paper title page; Figure 6