Kappa Labs public audit no. 2 · November 2026

"Over 80% agreement": what MT-Bench's famous judge number counts

MT-Bench is where "LLM as a judge" became standard practice. Its paper (Zheng et al., NeurIPS 2023) reports that GPT-4, judging which of two chatbot answers is better, agrees with expert humans "over 80%" of the time, the same as experts agree with each other. LMSYS released every vote behind that figure, 3,355 human judgments (2,668 from the expert raters behind the headline) and 2,400 GPT-4 verdicts across 80 questions, which is exactly what makes a check like this possible. Credit to them for that. The paper reports the answer-order effect itself (section 3.3) and states the tie rule under Table 5.

What we found

1. The published figures reproduce exactly. Every agreement rate and every vote count in the paper's Table 5 comes out of the released data to the vote.

2. Most of the 85% comes from which chatbots are being compared. MT-Bench pits strong models against much weaker ones. A rule that never reads the answers, and simply picks whichever chatbot the experts preferred on the other 79 questions, matches the experts 81% of the time on the same cases. GPT-4 reaches 85%, so reading the answers adds about 4 points (95% interval 2 to 6).

3. The 85% is measured on two thirds of the comparisons. The figure leaves out every case where either side called a tie. That includes the 380 of 2,400 cases (16%) where GPT-4's verdict changed once the two answers swapped places, which the paper counts as a tie. Together these remove 35% of the GPT-4 and expert comparisons, and they are cases the experts themselves agree on less. Where GPT-4's verdict flipped with the answer order, the experts agree with each other on 53% of pairs, against 68% where GPT-4 gave a clear verdict.

4. Scoring those cases, GPT-4 matches the experts' clear verdicts 70% of the time (95% interval 66% to 73%). That counts a tie or a self-contradiction from GPT-4 as a miss whenever the expert picked a winner. One expert matches another's clear verdict on exactly the same terms 70% of the time too.

5. "The same level as humans" holds on equal terms. The paper's table sets GPT-4's 85% beside the experts' 81%, but the two cover different cases. On the same cases, GPT-4 agrees with the experts 85% of the time and the experts agree with each other 83%: a gap of +1.6 points (95% interval −1.4 to +4.8).

Intervals resample whole questions, so they allow for the 80 questions each carrying many votes.

Why it matters

"Our judge agrees with humans X% of the time" is the sentence that sells most automated evaluations. MT-Bench shows the claim can be true and still say less than it seems: what matters is how many cases it was computed on, how the judge and the humans did on the rest, and how far the judge beats a rule that never looks at the answers. Those three numbers are worth asking for alongside any judge's agreement figure.

Kappa Labs runs independent audits of AI evaluation scores. Contact: hello@kappalabs.co.uk

Data: lmsys/mt_bench_human_judgments on Hugging Face, CC BY 4.0. Paper: Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023 Datasets and Benchmarks, arXiv 2306.05685 (v4, Table 5).