NUWA Evidence record
o1-preview
JudgeBench: A Benchmark for Evaluating LLM-based Judges · Evidence record
Original source charts and materials
Reported results
Source order · reported configurations
Select systems
Overall accuracy
0–100%
o1-previewValues describe the measured outcome under the source protocol, not a general safety score.
Table 2: accuracy of the Arena-Hard judge using different underlying models. This measures judging ability, not a risk rate. Evaluation date not specified; publication / record dates are shown below.
Evaluation conditions and system configurations
- o1-preview
- See the source protocol; no independent trial count inferred. · Table 2 · JudgeBench: A Benchmark for Evaluating LLM-based Judges
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
| System / configuration | Overall accuracy | Knowledge | Reasoning | Math | Coding | Publication / verification |
|---|---|---|---|---|---|---|
| o1-previewHarness not specified | 75.43% | 66.23% | 79.59% | 85.71% | 85.71% | Source checked2026-10-25 |
Method and scope
Evaluates model-based judges on challenging response comparisons to examine reliability of automated evaluation.
- Evidence type
- Benchmark
- Assessment subject
- Model or agent system in the original evaluation configuration
- Primary dimension
- Capability
- Evidence snapshot
- 2026-10-29.1
Classified by the operational outcome in the original protocol. Topic associations are documented separately from question-level matrix claims.
Source finding
Source-reported measurements for o1-preview in Table 2; values are transcribed, not independently reproduced.
Limitations
Judge performance is methodological context; it does not directly measure a frontier-risk capability.
Configuration, task and aggregation differences remain as reported. Source order is retained. Transcription is not independent replication; no cross-protocol ranking is supported.
Related research questions
References
Numerical results are reported by the cited authors.
- [1] Sijun Tan, Siyuan Zhuang, et al.. JudgeBench: A Benchmark for Evaluating LLM-based Judges. 2024-10-16. v2 [arXiv]; Author affiliations on the original paper title page; Table 2