{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"theagentcompany","title":"TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks","aliases":["TheAgentCompany"],"authors":[],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"theagentcompany-primary","attributionSourceId":"theagentcompany-primary","originalUrl":"https://arxiv.org/abs/2412.14161","publicationDate":"2024-12-18","updatedOn":"2026-10-25","areas":["autonomy"],"scopeNote":{"en":"Mapped to the documented research question. A risk-topic association is not a severity score.","zh":"依据明确研究问题关联；风险主题不代表严重性评分。"},"protocolIds":["theagentcompany"],"contributions":[],"summary":{"en":"How do agents complete consequential workplace tasks in a simulated company?","zh":"智能体如何在模拟公司中完成实际工作任务？"},"note":{"question":{"en":"How do agents complete consequential workplace tasks in a simulated company?","zh":"智能体如何在模拟公司中完成实际工作任务？"},"method":{"en":"Combines browsing, coding, file operations and interaction with simulated coworkers.","zh":"结合浏览、编程、文件操作和与模拟同事的交互。"},"finding":null,"supports":null,"limits":{"en":"Task success and evaluation-boundary behavior are separate outcomes; a broken environment is not an ordinary task failure.","zh":"任务成功与评测边界行为属于不同结果；环境故障不是普通任务失败。"},"connections":null},"noteAvailability":{"en":"This record describes the protocol. Report-specific outcomes are linked through assessments; no unsourced general finding is added.","zh":"本档案说明协议。报告特定结果通过评估记录关联，不补写无出处的总体发现。"},"resources":[],"tags":[]},"reading":{"id":"theagentcompany","title":"TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks","en":{"question":"Can agents perform diverse digital work in a simulated software company?","design":"A self-contained environment supplies internal sites, files, tools and coworkers; checkpoints award partial credit with a substantial full-completion bonus.","findings":["The scoring adds 50% credit only when every checkpoint is completed (§4.1).","Humans tested feasibility, but the authors did not collect a representative human-performance baseline (Appendix J)."],"interpretation":"NUWA interpretation: checkpoint-weighted progress and fully completed work are different outcomes.","limits":"Simulated coworkers and sites restrict workplace realism; task feasibility checks do not establish human–agent productivity comparisons.","decisionRelevance":"Model recommendation: report complete-task rate alongside partial score and avoid unsupported labor-substitution estimates.","takeaway":"NUWA interpretation: checkpoint-weighted progress and fully completed work are different outcomes.","connections":[],"scoring":"Checkpoint partial credit plus 50% credit awarded only for full completion.","conditions":"Self-contained simulated company; feasibility-tested but no representative human baseline.","statistics":"The scoring adds 50% credit only when every checkpoint is completed (§4.1).","usage":"Model recommendation: report complete-task rate alongside partial score and avoid unsupported labor-substitution estimates."},"zh":{"question":"智能体能否在模拟软件公司完成多样数字工作？","design":"封闭环境含内部网站、文件、工具及同事；检查点给部分分，全完成另有大额奖励。","findings":["仅完成全部检查点时加50%分（§4.1）。","人类验证可行性，但未收集代表性表现基线（附录J）。"],"interpretation":"NUWA 解读：检查点进度分与完整完成工作是不同结果。","limits":"模拟同事和网站限制职场真实性，可行性检查不能确立人机生产率比较。","decisionRelevance":"模型建议：并列完整完成率与部分分，不作无依据劳动替代估计。","takeaway":"NUWA 解读：检查点进度分与完整完成工作是不同结果。","connections":[],"scoring":"检查点部分分，全完成才额外计50%。","conditions":"封闭模拟公司，验证可行但无代表性人类表现基线。","statistics":"仅完成全部检查点时加50%分（§4.1）。","usage":"模型建议：并列完整完成率与部分分，不作无依据劳动替代估计。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"theagentcompany-primary","url":"https://arxiv.org/html/2412.14161","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2412.14161; sha256:b5a66c55f2181c8ab279c5bd6ceb0f92badc66ae8378c0dc12dcedee6204b876; archived source capture; exact capture time not independently established","artifactSha256":"b5a66c55f2181c8ab279c5bd6ceb0f92badc66ae8378c0dc12dcedee6204b876"},{"sourceId":"theagentcompany-primary","url":"https://arxiv.org/html/2412.14161","locator":"§4.1 Evaluation Metrics","version":"Selected original resource https://arxiv.org/html/2412.14161; sha256:b5a66c55f2181c8ab279c5bd6ceb0f92badc66ae8378c0dc12dcedee6204b876; archived source capture; exact capture time not independently established","artifactSha256":"b5a66c55f2181c8ab279c5bd6ceb0f92badc66ae8378c0dc12dcedee6204b876"},{"sourceId":"theagentcompany-primary","url":"https://arxiv.org/html/2412.14161","locator":"Appendix J Limitation","version":"Selected original resource https://arxiv.org/html/2412.14161; sha256:b5a66c55f2181c8ab279c5bd6ceb0f92badc66ae8378c0dc12dcedee6204b876; archived source capture; exact capture time not independently established","artifactSha256":"b5a66c55f2181c8ab279c5bd6ceb0f92badc66ae8378c0dc12dcedee6204b876"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"d309349115d503a0b02cacefe9728629c1eeacc7e93a6b1f2a92506a19216bb4"}},"sources":[{"id":"theagentcompany-primary","title":"TheAgentCompany","url":"https://arxiv.org/abs/2412.14161","version":"2026-10-25 source capture","publicationDate":"2024-12-18","datePrecision":"day","type":"original-report","verifiedOn":"2026-10-25","authors":[],"documentId":"arxiv:2412.14161"}]}