Risk Evaluation · Benchmark

JudgeBench: A Benchmark for Evaluating LLM-based Judges

NUWA research note · independent review pending

First published: 2024-10-16Source checked: 2026-10-25

NUWA Research Note

Research question Design and assumptions Source findings NUWA interpretation Limits and contrary evidence Research and decision relevance Institutions and roles Original resources

Model-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.

At a glance

NUWA interpretation: alignment with stylistic preference is different from judging substantive correctness.

Research question

Can LLM judges distinguish objectively correct from incorrect sophisticated responses?

Design and assumptions

Difficult knowledge, reasoning, math and coding datasets are converted to labeled response pairs, with both presentation orders and explicit tie handling.

Source findings

Many strong judges, including GPT-4o, performed only slightly above random in the reported evaluation.

Inconsistent order decisions or ties in both trials count as incorrect.

NUWA interpretation

NUWA interpretation: alignment with stylistic preference is different from judging substantive correctness.

Limits and contrary evidence

Curated correct/incorrect pairs do not cover all open-ended evaluation qualities; pair construction and reference correctness still matter.

Research and decision relevance

Model recommendation: validate the judge on task-relevant difficult pairs and report order consistency.

Precise source locations

JudgeBench: A Benchmark for Evaluating LLM-Based Judges · AbstractSelected original resource https://arxiv.org/html/2410.12784v2; sha256:953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1; archived source capture; exact capture time not independently established
JudgeBench: A Benchmark for Evaluating LLM-Based Judges · §3 JudgeBenchSelected original resource https://arxiv.org/html/2410.12784v2; sha256:953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1; archived source capture; exact capture time not independently established
JudgeBench: A Benchmark for Evaluating LLM-Based Judges · §4 EvaluationSelected original resource https://arxiv.org/html/2410.12784v2; sha256:953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1; archived source capture; exact capture time not independently established
JudgeBench: A Benchmark for Evaluating LLM-Based Judges · ConclusionSelected original resource https://arxiv.org/html/2410.12784v2; sha256:953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1; archived source capture; exact capture time not independently established

Source checking, extraction, editorial approval and replication are recorded separately. Methodology →

Institutions and roles

Research by University of California, Berkeley

Authors (8)

Sijun Tan · Siyuan Zhuang · Kyle Montgomery · William Y. Tang · Alejandro Cuadron · Chenguang Wang · Raluca Ada Popa · Ion Stoica

Evaluation protocols

JudgeBench: A Benchmark for Evaluating LLM-based Judges →

Cite and reuse

Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.

BibTeX ↓RIS ↓Record + source metadata · JSON ↓Reading coverage · JSON Suggest a correction

Original publication · BibTeX

@misc{judgebench,
  title = {JudgeBench: A Benchmark for Evaluating LLM-based Judges},
  url = {https://arxiv.org/abs/2410.12784v2},
  author = {Sijun Tan and Siyuan Zhuang and Kyle Montgomery and William Y. Tang and Alejandro Cuadron and Chenguang Wang and Raluca Ada Popa and Ion Stoica},
  year = {2024},
  date = {2024-10-16},
  version = {arXiv:2410.12784v2},
  note = {arXiv:2410.12784v2}
}

NUWA record · BibTeX

@misc{judgebench-nuwa,
  title = {JudgeBench: A Benchmark for Evaluating LLM-based Judges — NUWA research record},
  url = {https://whitzard.tech/en/nuwa/whitzard-index/work/judgebench/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--judgebench.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--judgebench.json.}
}

Evidence snapshot 2026-10-29.1 · Index updates · RSS

References

[1] Sijun Tan, Siyuan Zhuang, et al.. JudgeBench: A Benchmark for Evaluating LLM-based Judges. 2024-10-16. v2 [arXiv]; Author affiliations on the original paper title page; Abstract; §3 JudgeBench