{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"judgebench","title":"JudgeBench: A Benchmark for Evaluating LLM-based Judges","aliases":[],"authors":["Sijun Tan","Siyuan Zhuang","Kyle Montgomery","William Y. Tang","Alejandro Cuadron","Chenguang Wang","Raluca Ada Popa","Ion Stoica"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"judgebench-publication","attributionSourceId":"judgebench-publication-author-source","originalUrl":"https://arxiv.org/abs/2410.12784v2","publicationDate":"2024-10-16","updatedOn":"2026-10-25","areas":[],"scopeNote":{"en":"Relevance is assigned to the research question and stated conditions of the work. A topic tag is not a risk rating.","zh":"风险关联依据研究问题及明确条件建立，主题标签不代表风险评级。"},"protocolIds":["judgebench"],"contributions":[{"organizationId":"berkeley","role":"research","sourceId":"judgebench-publication-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Evaluates model-based judges on challenging response comparisons to examine reliability of automated evaluation.","zh":"通过困难回答比较，评测模型评判器并考察自动评测可靠性。"},"note":{"question":{"en":"Can a model judge reliably choose a correct response over a plausible but incorrect response on difficult questions?","zh":"在困难问题上，模型裁判能否可靠地选出正确回答，而不是看似合理但错误的回答？"},"method":{"en":"The pipeline samples multiple responses from one strong model on tasks with verifiable answers, then pairs a correct and an incorrect response. Pairs are judged in both orders across knowledge, reasoning, mathematics and coding tasks.","zh":"该流程在有可验证答案的任务上从同一个较强模型采样多个回答，再将正确与错误回答配对。裁判按两种顺序评判这些回答对，涵盖知识、推理、数学和编码任务。"},"finding":{"en":"The benchmark exposes substantial difficulty for LLM judges on correctness-based distinctions, including judges that do well on preference-oriented benchmarks. The source analyzes judge choice, response order and dataset-construction effects.","zh":"该基准揭示了模型裁判判断正确性时的明显困难，包括在偏好导向基准上表现较好的裁判。来源分析了裁判选择、回答顺序和数据构造的影响。"},"supports":{"en":"A fluent or preferred answer is not automatically a valid ground-truth judgment. Judge capability should be checked on the actual kind of correctness distinction an evaluation needs.","zh":"流畅或更受偏好的回答并不自动构成有效的真值判断。应针对评估实际需要区分的正确性类型检查裁判能力。"},"limits":{"en":"Filtering for questions where a response-generating model produces both correct and incorrect answers induces selection bias. Verifier quality and the chosen source tasks bound what the benchmark measures; the work does not validate every safety-evaluation judge.","zh":"筛选某个回答生成模型同时产生正确与错误答案的问题，会引入选择偏差。验证器质量和所选原始任务限制了基准的测量范围；该研究并未验证所有安全评估裁判。"},"connections":null},"resources":[],"noteStatus":{"question":"verified","method":"verified","finding":"verified","supports":"verified","limits":"verified","connections":"limited"},"noteAvailability":{"en":"The substantive sections shown here were checked against the original source. Unfilled sections are omitted; source checking is not experimental replication.","zh":"此处展示的实质性章节已对照原始来源核查；未完成的章节已省略。来源核查不等于实验复现。"},"noteSourceLocator":"arXiv:2410.12784v2, §§3, 4.1–4.4; Appendix A.3–A.5","tags":[]},"reading":{"id":"judgebench","title":"JudgeBench: A Benchmark for Evaluating LLM-based Judges","en":{"question":"Can LLM judges distinguish objectively correct from incorrect sophisticated responses?","design":"Difficult knowledge, reasoning, math and coding datasets are converted to labeled response pairs, with both presentation orders and explicit tie handling.","findings":["Many strong judges, including GPT-4o, performed only slightly above random in the reported evaluation.","Inconsistent order decisions or ties in both trials count as incorrect."],"interpretation":"NUWA interpretation: alignment with stylistic preference is different from judging substantive correctness.","limits":"Curated correct/incorrect pairs do not cover all open-ended evaluation qualities; pair construction and reference correctness still matter.","decisionRelevance":"Model recommendation: validate the judge on task-relevant difficult pairs and report order consistency.","takeaway":"NUWA interpretation: alignment with stylistic preference is different from judging substantive correctness.","connections":[],"scoring":"Correct preference for the objective correct answer across two orderings; inconsistent/two-tie results fail.","conditions":null,"statistics":"Many strong judges, including GPT-4o, performed only slightly above random in the reported evaluation.","usage":"Model recommendation: validate the judge on task-relevant difficult pairs and report order consistency."},"zh":{"question":"模型判官能否区分复杂回答的客观正误？","design":"知识、推理、数学和编程难题转为标签配对，双顺序测试并明确平局处理。","findings":["所报评价中许多强判官含GPT-4o仅略高随机。","顺序不一致或两次都平局记错。"],"interpretation":"NUWA 解读：风格偏好一致不同于实质正确性判断。","limits":"构造正误配对不能覆盖全部开放评价品质，配对及参考正确性仍重要。","decisionRelevance":"模型建议：用相关难题验证判官，并报告顺序一致性。","takeaway":"NUWA 解读：风格偏好一致不同于实质正确性判断。","connections":[],"scoring":"双顺序中选客观正确回答，不一致或两次平局为错。","conditions":null,"statistics":"所报评价中许多强判官含GPT-4o仅略高随机。","usage":"模型建议：用相关难题验证判官，并报告顺序一致性。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"judgebench-publication-author-source","url":"https://arxiv.org/html/2410.12784v2","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2410.12784v2; sha256:953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1; archived source capture; exact capture time not independently established","artifactSha256":"953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1"},{"sourceId":"judgebench-publication-author-source","url":"https://arxiv.org/html/2410.12784v2","locator":"§3 JudgeBench","version":"Selected original resource https://arxiv.org/html/2410.12784v2; sha256:953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1; archived source capture; exact capture time not independently established","artifactSha256":"953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1"},{"sourceId":"judgebench-publication-author-source","url":"https://arxiv.org/html/2410.12784v2","locator":"§4 Evaluation","version":"Selected original resource https://arxiv.org/html/2410.12784v2; sha256:953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1; archived source capture; exact capture time not independently established","artifactSha256":"953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1"},{"sourceId":"judgebench-publication-author-source","url":"https://arxiv.org/html/2410.12784v2","locator":"Conclusion","version":"Selected original resource https://arxiv.org/html/2410.12784v2; sha256:953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1; archived source capture; exact capture time not independently established","artifactSha256":"953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"a8c970104f7565bd749811de983f70a3337c0cc08cf501366993b581d266ef8e"}},"sources":[{"id":"judgebench-publication","title":"JudgeBench: A Benchmark for Evaluating LLM-based Judges","url":"https://arxiv.org/abs/2410.12784v2","version":"arXiv:2410.12784v2","publicationDate":"2024-10-16","datePrecision":"day","type":"paper","verifiedOn":"2026-10-25","documentId":"arxiv:2410.12784","authors":["Sijun Tan","Siyuan Zhuang","Kyle Montgomery","William Y. Tang","Alejandro Cuadron","Chenguang Wang","Raluca Ada Popa","Ion Stoica"],"aliases":["https://doi.org/10.48550/arXiv.2410.12784","https://arxiv.org/abs/2410.12784","https://arxiv.org/pdf/2410.12784"]}]}