Risk Evaluation · Benchmark

TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

NUWA research note · independent review pending

First published: 2024-12-18Source checked: 2026-10-25

Original source ↗Cite this record ↓Autonomy →

NUWA Research Note

Research question Design and assumptions Source findings NUWA interpretation Limits and contrary evidence Research and decision relevance Institutions and roles Original resources

Model-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.

At a glance

NUWA interpretation: checkpoint-weighted progress and fully completed work are different outcomes.

Research question

Can agents perform diverse digital work in a simulated software company?

Design and assumptions

A self-contained environment supplies internal sites, files, tools and coworkers; checkpoints award partial credit with a substantial full-completion bonus.

Source findings

The scoring adds 50% credit only when every checkpoint is completed (§4.1).

Humans tested feasibility, but the authors did not collect a representative human-performance baseline (Appendix J).

NUWA interpretation

NUWA interpretation: checkpoint-weighted progress and fully completed work are different outcomes.

Limits and contrary evidence

Simulated coworkers and sites restrict workplace realism; task feasibility checks do not establish human–agent productivity comparisons.

Research and decision relevance

Model recommendation: report complete-task rate alongside partial score and avoid unsupported labor-substitution estimates.

Precise source locations

TheAgentCompany · AbstractSelected original resource https://arxiv.org/html/2412.14161; sha256:b5a66c55f2181c8ab279c5bd6ceb0f92badc66ae8378c0dc12dcedee6204b876; archived source capture; exact capture time not independently established
TheAgentCompany · §4.1 Evaluation MetricsSelected original resource https://arxiv.org/html/2412.14161; sha256:b5a66c55f2181c8ab279c5bd6ceb0f92badc66ae8378c0dc12dcedee6204b876; archived source capture; exact capture time not independently established
TheAgentCompany · Appendix J LimitationSelected original resource https://arxiv.org/html/2412.14161; sha256:b5a66c55f2181c8ab279c5bd6ceb0f92badc66ae8378c0dc12dcedee6204b876; archived source capture; exact capture time not independently established

Source checking, extraction, editorial approval and replication are recorded separately. Methodology →

Institutions and roles

Affiliation not specified

Evaluation protocols

TheAgentCompany →

Cite and reuse

Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.

BibTeX ↓RIS ↓Record + source metadata · JSON ↓Reading coverage · JSON Suggest a correction

Original publication · BibTeX

@misc{theagentcompany,
  title = {TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks},
  url = {https://arxiv.org/abs/2412.14161},
  year = {2024},
  date = {2024-12-18},
  version = {2026-10-25 source capture},
  note = {2026-10-25 source capture}
}

NUWA record · BibTeX

@misc{theagentcompany-nuwa,
  title = {TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks — NUWA research record},
  url = {https://whitzard.tech/en/nuwa/whitzard-index/work/theagentcompany/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--theagentcompany.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--theagentcompany.json.}
}

Evidence snapshot 2026-10-29.1 · Index updates · RSS

References

[1] TheAgentCompany. 2024-12-18 ; Abstract; §4.1 Evaluation Metrics; Appendix J Limitation