NUWA Evaluation dossier
JudgeBench: A Benchmark for Evaluating LLM-based Judges
Evaluates model-based judges on challenging response comparisons to examine reliability of automated evaluation.
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
JudgeBench: A Benchmark for Evaluating LLM-based Judges · Overall accuracy
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
15 observations · 0 dated
● Matched◇ Configuration differs□ Some conditions undisclosed
15 observations lack a verified model release date. They remain in Data and Systems.
Select a point or table row to inspect its conditions and provenance.
15 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
 o3-mini (high) | 80.86%Overall accuracy | Incomplete | Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges— | Not disclosedJudgeBench: A Benchmark for Evaluating LLM-Based Judges | Release —Report — |
| o3-mini (medium) | 76.57%Overall accuracy | Incomplete | Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges— | Not disclosedJudgeBench: A Benchmark for Evaluating LLM-Based Judges | Release —Report — |
| o1-preview | 75.43%Overall accuracy | Incomplete | Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges— | Not disclosedJudgeBench: A Benchmark for Evaluating LLM-Based Judges | Release —Report — |
 Deepseek-R1 | 73.14%Overall accuracy | Incomplete | Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges— | Not disclosedJudgeBench: A Benchmark for Evaluating LLM-Based Judges | Release —Report — |
| o3-mini (low) | 70.57%Overall accuracy | Incomplete | Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges— | Not disclosedJudgeBench: A Benchmark for Evaluating LLM-Based Judges | Release —Report — |
| o1-mini | 65.71%Overall accuracy | Incomplete | Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges— | Not disclosedJudgeBench: A Benchmark for Evaluating LLM-Based Judges | Release —Report — |
 Claude-3.5-Sonnet | 64.29%Overall accuracy | Incomplete | Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges— | Not disclosedJudgeBench: A Benchmark for Evaluating LLM-Based Judges | Release —Report — |
 Llama-3.1-405B-Instruct | 56.86%Overall accuracy | Incomplete | Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges— | Not disclosedJudgeBench: A Benchmark for Evaluating LLM-Based Judges | Release —Report — |
| GPT-4o | 56.57%Overall accuracy | Incomplete | Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges— | Not disclosedJudgeBench: A Benchmark for Evaluating LLM-Based Judges | Release —Report — |
| Llama-3.1-70B-Instruct | 52.29%Overall accuracy | Incomplete | Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges— | Not disclosedJudgeBench: A Benchmark for Evaluating LLM-Based Judges | Release —Report — |
| GPT-4o-mini | 50%Overall accuracy | Incomplete | Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges— | Not disclosedJudgeBench: A Benchmark for Evaluating LLM-Based Judges | Release —Report — |
 Gemini-1.5-pro | 47.14%Overall accuracy | Incomplete | Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges— | Not disclosedJudgeBench: A Benchmark for Evaluating LLM-Based Judges | Release —Report — |
| Llama-3.1-8B-Instruct | 40.86%Overall accuracy | Incomplete | Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges— | Not disclosedJudgeBench: A Benchmark for Evaluating LLM-Based Judges | Release —Report — |
| Gemini-1.5-flash | 39.71%Overall accuracy | Incomplete | Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges— | Not disclosedJudgeBench: A Benchmark for Evaluating LLM-Based Judges | Release —Report — |
| Claude-3-Haiku | 33.14%Overall accuracy | Incomplete | Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges— | Not disclosedJudgeBench: A Benchmark for Evaluating LLM-Based Judges | Release —Report — |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- Model or agent system in the original evaluation configuration
- Measurement target
- Evaluates model-based judges on challenging response comparisons to examine reliability of automated evaluation.
- Scoring / outcome
- Overall accuracy · Table 2: accuracy of the Arena-Hard judge using different underlying models. This measures judging ability, not a risk rate.
- Interpretation boundary
- Judge performance is methodological context; it does not directly measure a frontier-risk capability.
The pipeline samples multiple responses from one strong model on tasks with verifiable answers, then pairs a correct and an incorrect response. Pairs are judged in both orders across knowledge, reasoning, mathematics and coding tasks.
A reported evaluation
The source evaluates GPT-4o · Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges.
Reported measurement: Overall accuracy · 56.57%.
Table 2: accuracy of the Arena-Hard judge using different underlying models. This measures judging ability, not a risk rate. See the source protocol; no independent trial count inferred.
Source-reported measurements for GPT-4o in Table 2; values are transcribed, not independently reproduced.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can LLM judges distinguish objectively correct from incorrect sophisticated responses?
- Task population
- Difficult knowledge, reasoning, math and coding datasets are converted to labeled response pairs, with both presentation orders and explicit tie handling.
- Scoring and failure modes
- Correct preference for the objective correct answer across two orderings; inconsistent/two-tie results fail.
- System conditions
- Not established by the retained material
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: Many strong judges, including GPT-4o, performed only slightly above random in the reported evaluation.
- Validity and interpretation
- Curated correct/incorrect pairs do not cover all open-ended evaluation qualities; pair construction and reference correctness still matter.
- Appropriate research use
- Model recommendation: validate the judge on task-relevant difficult pairs and report order consistency.
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges · AbstractSelected original resource https://arxiv.org/html/2410.12784v2; sha256:953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1; archived source capture; exact capture time not independently established
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges · §3 JudgeBenchSelected original resource https://arxiv.org/html/2410.12784v2; sha256:953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1; archived source capture; exact capture time not independently established
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges · §4 EvaluationSelected original resource https://arxiv.org/html/2410.12784v2; sha256:953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1; archived source capture; exact capture time not independently established
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges · ConclusionSelected original resource https://arxiv.org/html/2410.12784v2; sha256:953e402aa5d7e59190cf34eebdd65bab720fab389e3d10bd1201950acb62bfd1; archived source capture; exact capture time not independently established
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
JudgeBench: A Benchmark for Evaluating LLM-Based Judges
JudgeBench: A Benchmark for Evaluating LLM-Based Judges
Actual evaluators: Not disclosed
15 result records · 15 reported system names
Protocols and comparison decisions (5)
JudgeBench: A Benchmark for Evaluating LLM-based Judges: Coding · JudgeBench: A Benchmark for Evaluating LLM-Based Judgessource-report · Scope reviewed
JudgeBench: A Benchmark for Evaluating LLM-based Judges: Knowledge · JudgeBench: A Benchmark for Evaluating LLM-Based Judgessource-report · Scope reviewed
JudgeBench: A Benchmark for Evaluating LLM-based Judges: Math · JudgeBench: A Benchmark for Evaluating LLM-Based Judgessource-report · Scope reviewed
Original source charts and materials
Reported results
Source order · reported configurations
Select systems
Overall accuracy
0–100%GPT-4oGPT-4o · 56.57% · Table 2: accuracy of the Arena-Hard judge using different underlying models. This measures judging ability, not a risk rate. · See the source protocol; no independent trial count inferred.
56.57%GPT-4o-miniGPT-4o-mini · 50.00% · Table 2: accuracy of the Arena-Hard judge using different underlying models. This measures judging ability, not a risk rate. · See the source protocol; no independent trial count inferred.
50.00%o1-previewo1-preview · 75.43% · Table 2: accuracy of the Arena-Hard judge using different underlying models. This measures judging ability, not a risk rate. · See the source protocol; no independent trial count inferred.
75.43%o1-minio1-mini · 65.71% · Table 2: accuracy of the Arena-Hard judge using different underlying models. This measures judging ability, not a risk rate. · See the source protocol; no independent trial count inferred.
65.71%o3-mini (high)o3-mini (high) · 80.86% · Table 2: accuracy of the Arena-Hard judge using different underlying models. This measures judging ability, not a risk rate. · See the source protocol; no independent trial count inferred.
80.86%o3-mini (medium)o3-mini (medium) · 76.57% · Table 2: accuracy of the Arena-Hard judge using different underlying models. This measures judging ability, not a risk rate. · See the source protocol; no independent trial count inferred.
76.57%o3-mini (low)o3-mini (low) · 70.57% · Table 2: accuracy of the Arena-Hard judge using different underlying models. This measures judging ability, not a risk rate. · See the source protocol; no independent trial count inferred.
70.57%Claude-3.5-SonnetClaude-3.5-Sonnet · 64.29% · Table 2: accuracy of the Arena-Hard judge using different underlying models. This measures judging ability, not a risk rate. · See the source protocol; no independent trial count inferred.
64.29%Claude-3-HaikuClaude-3-Haiku · 33.14% · Table 2: accuracy of the Arena-Hard judge using different underlying models. This measures judging ability, not a risk rate. · See the source protocol; no independent trial count inferred.
33.14%Llama-3.1-405B-InstructLlama-3.1-405B-Instruct · 56.86% · Table 2: accuracy of the Arena-Hard judge using different underlying models. This measures judging ability, not a risk rate. · See the source protocol; no independent trial count inferred.
56.86%Llama-3.1-70B-InstructLlama-3.1-70B-Instruct · 52.29% · Table 2: accuracy of the Arena-Hard judge using different underlying models. This measures judging ability, not a risk rate. · See the source protocol; no independent trial count inferred.
52.29%Llama-3.1-8B-InstructLlama-3.1-8B-Instruct · 40.86% · Table 2: accuracy of the Arena-Hard judge using different underlying models. This measures judging ability, not a risk rate. · See the source protocol; no independent trial count inferred.
40.86%Showing 12 of 15 systems
Values describe the measured outcome under the source protocol, not a general safety score.
Table 2: accuracy of the Arena-Hard judge using different underlying models. This measures judging ability, not a risk rate. Evaluation date not specified; publication / record dates are shown below.
Evaluation conditions and system configurations
- GPT-4o
- See the source protocol; no independent trial count inferred. · Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- GPT-4o-mini
- See the source protocol; no independent trial count inferred. · Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- o1-preview
- See the source protocol; no independent trial count inferred. · Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- o1-mini
- See the source protocol; no independent trial count inferred. · Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- o3-mini (high)
- See the source protocol; no independent trial count inferred. · Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- o3-mini (medium)
- See the source protocol; no independent trial count inferred. · Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- o3-mini (low)
- See the source protocol; no independent trial count inferred. · Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Claude-3.5-Sonnet
- See the source protocol; no independent trial count inferred. · Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Claude-3-Haiku
- See the source protocol; no independent trial count inferred. · Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Llama-3.1-405B-Instruct
- See the source protocol; no independent trial count inferred. · Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Llama-3.1-70B-Instruct
- See the source protocol; no independent trial count inferred. · Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Llama-3.1-8B-Instruct
- See the source protocol; no independent trial count inferred. · Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Gemini-1.5-pro
- See the source protocol; no independent trial count inferred. · Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Gemini-1.5-flash
- See the source protocol; no independent trial count inferred. · Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Deepseek-R1
- See the source protocol; no independent trial count inferred. · Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
| System / configuration | Overall accuracy | Knowledge | Reasoning | Math | Coding | Publication / verification |
|---|---|---|---|---|---|---|
| GPT-4oHarness not specified | 56.57% | 50.65% | 54.08% | 75.00% | 59.52% | Source checked2026-10-25 |
| GPT-4o-miniHarness not specified | 50.00% | 48.05% | 43.88% | 69.64% | 45.24% | Source checked2026-10-25 |
| o1-previewHarness not specified | 75.43% | 66.23% | 79.59% | 85.71% | 85.71% | Source checked2026-10-25 |
| o1-miniHarness not specified | 65.71% | 58.44% | 62.24% | 82.14% | 78.57% | Source checked2026-10-25 |
| o3-mini (high)Harness not specified | 80.86% | 67.53% | 89.80% | 87.50% | 100.0% | Source checked2026-10-25 |
| o3-mini (medium)Harness not specified | 76.57% | 62.34% | 86.73% | 85.71% | 92.86% | Source checked2026-10-25 |
| o3-mini (low)Harness not specified | 70.57% | 62.99% | 69.39% | 83.93% | 83.33% | Source checked2026-10-25 |
| Claude-3.5-SonnetHarness not specified | 64.29% | 62.34% | 66.33% | 66.07% | 64.29% | Source checked2026-10-25 |
| Claude-3-HaikuHarness not specified | 33.14% | 35.06% | 34.69% | 33.93% | 21.43% | Source checked2026-10-25 |
| Llama-3.1-405B-InstructHarness not specified | 56.86% | 55.84% | 54.08% | 69.64% | 50.00% | Source checked2026-10-25 |
| Llama-3.1-70B-InstructHarness not specified | 52.29% | 51.30% | 48.98% | 60.71% | 52.38% | Source checked2026-10-25 |
| Llama-3.1-8B-InstructHarness not specified | 40.86% | 38.31% | 45.92% | 44.64% | 33.33% | Source checked2026-10-25 |
| Gemini-1.5-proHarness not specified | 47.14% | 49.35% | 42.86% | 64.29% | 26.19% | Source checked2026-10-25 |
| Gemini-1.5-flashHarness not specified | 39.71% | 42.86% | 36.73% | 50.00% | 21.43% | Source checked2026-10-25 |
| Deepseek-R1Harness not specified | 73.14% | 59.09% | 82.65% | 80.36% | 92.86% | Source checked2026-10-25 |
Related research questions
References
Numerical results are reported by the cited authors.
- [1] Sijun Tan, Siyuan Zhuang, et al.. JudgeBench: A Benchmark for Evaluating LLM-Based Judges. 2024-10-16. v2 [arXiv]; Author affiliations on the original paper title page; Table 2