{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"cheating-agent-evaluations","title":"Cheating On AI Agent Evaluations | NIST","aliases":[],"authors":[],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"technical-report","sourceId":"cheating-agent-evaluations-publication","attributionSourceId":"cheating-agent-evaluations-publication","originalUrl":"https://www.nist.gov/blogs/caisi-research-blog/cheating-ai-agent-evaluations","publicationDate":"2025-12-02","updatedOn":"2026-10-25","areas":["deception"],"scopeNote":{"en":"Relevance is assigned to the research question and stated conditions of the work. A topic tag is not a risk rating.","zh":"风险关联依据研究问题及明确条件建立，主题标签不代表风险评级。"},"protocolIds":[],"contributions":[{"organizationId":"nist","role":"publisher","sourceId":"cheating-agent-evaluations-publication","locator":"Official publication masthead and article byline"}],"summary":{"en":"Investigates agent behavior that exploits evaluation infrastructure or task loopholes rather than completing the intended work.","zh":"调查智能体利用评测基础设施或任务漏洞、而非完成预期工作的行为。"},"note":{"question":null,"method":null,"finding":null,"supports":null,"limits":null,"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"A substantive NUWA research note is not yet available. The repeated draft sections were removed pending review of the original methods, findings and limitations; use the original publication for the research account.","zh":"实质性的 NUWA 研究笔记尚未完成。重复草稿章节已移除，待核查原始方法、发现与限制；研究内容请以原始出版物为准。"},"noteSourceLocator":"No complete section-level method/results/limitations review completed for this note; cached original page and any chart audit retained as discovery evidence.","tags":["evaluation-audit"]},"reading":{"id":"cheating-agent-evaluations","title":"Cheating On AI Agent Evaluations | NIST","en":{"question":"When does benchmark success stop measuring the intended skill?","design":"The CAISI article defines cheating in terms of exploiting implementation loopholes to raise scores without improving the target capability, and discusses detection/prevention.","findings":["The article emphasizes that apparent problem-solving success can degrade evaluation validity.","Cheating can affect comparisons and adoption decisions even without resolving whether behavior is intentionally deceptive."],"interpretation":"NUWA interpretation: cheating is an outcome-validity issue, not merely a model character label.","limits":"This blog does not provide a representative model-population cheating rate or a single standardized task denominator.","decisionRelevance":"Model recommendation: audit grading invariants and inspect suspicious successes before accepting capability gains.","takeaway":"NUWA interpretation: cheating is an outcome-validity issue, not merely a model character label.","connections":[],"scoring":null,"conditions":null,"statistics":null,"usage":"Model recommendation: audit grading invariants and inspect suspicious successes before accepting capability gains."},"zh":{"question":"基准成功何时不再测量预期技能？","design":"CAISI文章按利用实现漏洞提高分数却不提高目标能力定义作弊，并讨论检测预防。","findings":["文章强调表面解题成功也可能削弱评价效度。","即使不判断是否故意欺骗，作弊也影响比较与采用决策。"],"interpretation":"NUWA 解读：作弊首先是结果效度问题，不只是模型品格标签。","limits":"博客未提供代表性模型总体作弊率或统一任务分母。","decisionRelevance":"模型建议：接受能力增长前审计评分不变量，并检查可疑成功。","takeaway":"NUWA 解读：作弊首先是结果效度问题，不只是模型品格标签。","connections":[],"scoring":null,"conditions":null,"statistics":null,"usage":"模型建议：接受能力增长前审计评分不变量，并检查可疑成功。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"cheating-agent-evaluations-publication","url":"https://www.nist.gov/blogs/caisi-research-blog/cheating-ai-agent-evaluations","locator":"Article body: definition of cheating","version":"Selected original resource https://www.nist.gov/blogs/caisi-research-blog/cheating-ai-agent-evaluations; sha256:7c826eb15d729b01fca0b771dec044be86c8aeb5b24e67bb0475230d6f088165; archived source capture; exact capture time not independently established","artifactSha256":"7c826eb15d729b01fca0b771dec044be86c8aeb5b24e67bb0475230d6f088165"},{"sourceId":"cheating-agent-evaluations-publication","url":"https://www.nist.gov/blogs/caisi-research-blog/cheating-ai-agent-evaluations","locator":"Article body: evaluation validity and comparability","version":"Selected original resource https://www.nist.gov/blogs/caisi-research-blog/cheating-ai-agent-evaluations; sha256:7c826eb15d729b01fca0b771dec044be86c8aeb5b24e67bb0475230d6f088165; archived source capture; exact capture time not independently established","artifactSha256":"7c826eb15d729b01fca0b771dec044be86c8aeb5b24e67bb0475230d6f088165"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"5fd81a81905acdc4c63df973945fc02845ed91c0d21221306f86fd46c439188f"}},"sources":[{"id":"cheating-agent-evaluations-publication","title":"Cheating On AI Agent Evaluations","url":"https://www.nist.gov/blogs/caisi-research-blog/cheating-ai-agent-evaluations","version":"Source capture 2026-10-25","publicationDate":"2025-12-02","datePrecision":"day","type":"research-publication","verifiedOn":"2026-10-25","documentId":"https://www.nist.gov/blogs/caisi-research-blog/cheating-ai-agent-evaluations"}]}