风险评测 · 评测基准

JudgeBench: A Benchmark for Evaluating LLM-based Judges

NUWA 研究解读 · 独立复核待完成

首次发表: 2024-10-16来源核验: 2026-10-25

原始来源 ↗引用本记录 ↓

NUWA Research Note

研究问题研究设计与假设来源发现 NUWA 解读局限与相反证据研究与决策意义机构与角色原始资源

模型辅助整理 · 选读原始全文或官方文章中的相关章节；尚未完成逐章节审计。

研究速读

NUWA 解读：风格偏好一致不同于实质正确性判断。

研究问题

模型判官能否区分复杂回答的客观正误？

研究设计与假设

知识、推理、数学和编程难题转为标签配对，双顺序测试并明确平局处理。

来源发现

所报评价中许多强判官含GPT-4o仅略高随机。

顺序不一致或两次都平局记错。

NUWA 解读

NUWA 解读：风格偏好一致不同于实质正确性判断。

局限与相反证据

构造正误配对不能覆盖全部开放评价品质，配对及参考正确性仍重要。

研究与决策意义

模型建议：用相关难题验证判官，并报告顺序一致性。

精确来源位置

JudgeBench: A Benchmark for Evaluating LLM-Based Judges · AbstractSelected original resource https://arxiv.org/html/2410.12784v2; sha256:953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1; archived source capture; exact capture time not independently established
JudgeBench: A Benchmark for Evaluating LLM-Based Judges · §3 JudgeBenchSelected original resource https://arxiv.org/html/2410.12784v2; sha256:953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1; archived source capture; exact capture time not independently established
JudgeBench: A Benchmark for Evaluating LLM-Based Judges · §4 EvaluationSelected original resource https://arxiv.org/html/2410.12784v2; sha256:953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1; archived source capture; exact capture time not independently established
JudgeBench: A Benchmark for Evaluating LLM-Based Judges · ConclusionSelected original resource https://arxiv.org/html/2410.12784v2; sha256:953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1; archived source capture; exact capture time not independently established

来源核查、数据提取、编辑批准与实验复现分别记录。方法说明 →

机构与角色

研究机构加州大学伯克利分校

作者 (8)

Sijun Tan · Siyuan Zhuang · Kyle Montgomery · William Y. Tang · Alejandro Cuadron · Chenguang Wang · Raluca Ada Popa · Ion Stoica

评测协议

JudgeBench: A Benchmark for Evaluating LLM-based Judges →

引用与复用

引用研究发现时请引用原始出版物；引用 NUWA 的整理与解释时，单独引用本站记录及对应快照。

BibTeX ↓RIS ↓记录与来源元数据 · JSON ↓阅读覆盖 · JSON 提交更正

原始出版物 · BibTeX

@misc{judgebench,
  title = {JudgeBench: A Benchmark for Evaluating LLM-based Judges},
  url = {https://arxiv.org/abs/2410.12784v2},
  author = {Sijun Tan and Siyuan Zhuang and Kyle Montgomery and William Y. Tang and Alejandro Cuadron and Chenguang Wang and Raluca Ada Popa and Ion Stoica},
  year = {2024},
  date = {2024-10-16},
  version = {arXiv:2410.12784v2},
  note = {arXiv:2410.12784v2}
}

NUWA 记录 · BibTeX

@misc{judgebench-nuwa,
  title = {JudgeBench: A Benchmark for Evaluating LLM-based Judges — NUWA research record},
  url = {https://whitzard.tech/nuwa/whitzard-index/work/judgebench/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--judgebench.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--judgebench.json.}
}

证据快照 2026-10-29.1 · 索引更新 · RSS

参考文献

[1] Sijun Tan, Siyuan Zhuang 等. JudgeBench: A Benchmark for Evaluating LLM-based Judges. 2024-10-16. v2 [arXiv]; Author affiliations on the original paper title page; Abstract; §3 JudgeBench