Kappa Labs public audit no. 3 · November 2026
JudgmentBench, checked: the headline holds, and length is the open question
JudgmentBench, from Russell Yang, Ruishi Chen, Pierce Kelaita, Riya Ranjan, Sibo Ma, Charles Dickens, Matthew Guillod, Megan Ma and Julian Nyarko (arXiv 2605.25240, v3, June 2026), asks a question every legal AI team needs answered: when lawyers grade AI-written legal work, is it better to score each answer against a rubric or to ask which of two answers is better? The team built answers to 30 legal tasks at three known quality levels (intermediate, good and excellent), had 51 lawyers grade them both ways, and released everything under MIT, including eight AI judges' verdicts and a follow-up study. That openness is what made this audit possible, and it is rare. We thank them for it.
The headline reproduces exactly. From the released files we get the same four numbers the paper reports:
| Paper | Our check | |
|---|---|---|
| Rank agreement with the designed quality order, comparisons | 0.908 | 0.908 |
| Rank agreement, rubrics | 0.150 | 0.150 |
| Lawyers pick the better answer, comparisons | 66.9% | 66.9% |
| Lawyers pick the better answer, rubrics | 54.2% | 54.2% |
Comparison beats rubrics, as the paper says.
What we add: in this benchmark, the better answer is almost always the longer one. The median answer runs 932 words at the intermediate level, 1,396 at good and 2,000 at excellent. In 1,519 of the 1,530 lawyer comparisons (99.3%), the higher-quality answer is also the longer one. So a rule that never reads the answers, and simply picks the longer one, recovers the designed order 99.3% of the time. For reference, the lawyers pick the better answer 66.9% of the time and the eight AI judges between 59.9% and 89.0%:
| Judge | Picks the better answer |
|---|---|
| Always pick the longer answer | 99.3% |
| Claude Opus 4.8 | 89.0% |
| GLM 5.2 | 87.8% |
| Claude Sonnet 5 | 85.6% |
| Qwen 3.7 Max | 84.1% |
| Gemini 3.6 Flash | 82.6% |
| Gemini 3.1 Pro | 82.4% |
| GPT-5.4-mini | 70.7% |
| Lawyers | 66.9% |
| GPT-5.4 | 59.9% |
This does not mean any judge, human or AI, is choosing by length. Because quality and length move together in almost every pair, the released comparisons cannot tell the two apart. A judge rewarding quality and a judge rewarding length would score the same.
The authors had already started testing this. The dataset includes a follow-up released with it after the paper (not in arXiv v3): 300 comparisons in which the weaker answer was rewritten to be longer. Before the change, the excellent answer won 211 of 300 (70%, interval 59% to 81%). After it, the excellent answer won 131 of 300 (44%, interval 36% to 52%), a fall of 27 points (interval 11 to 41 points, 12 annotators). The fall shows up against both weaker levels: against intermediate answers the excellent one went from 134 to 77 wins out of 180, and against good answers from 77 to 54 out of 120. The authors note, fairly, that rewriting an answer changes more than its length, so this study alone cannot separate length from content. We agree, and it is a strong first step.
Where this leaves the headline. The comparison-over-rubrics result is real in this data. What the data cannot yet say is how much of the lawyers' and AI judges' success comes from recognising quality and how much from the fact that better answers here are longer. A version of the benchmark with quality and length set independently would settle it, and the authors' follow-up shows they are well placed to build one. We would be glad to run the same check on it.
Figures from the released dataset (Hugging Face judgmentbench/JudgmentBench, revision e4ae23e, MIT). Intervals are 95%. Kappa Labs checks published AI evaluation results; we report results, not methods. hello@kappalabs.co.uk