A capstone study of LLM-as-a-judge
Do LLM judges know when they're wrong?
Three LLM judges compared pairs of chatbot answers from MT-Bench, and every verdict was checked against what human voters chose. Here is what they said, and how sure they were.
Qwen2.5-7B's stated confidence, against how often it matched the human majority.
of its verdicts flip when the same two answers simply swap places.
of its wrong verdicts would pass a pipeline that auto-accepts anything stated at ≥ 90%.
01 · Trick the judge
Same two answers. Opposite verdict. Same confidence.
Swap which answer is shown first, or pad both with filler that adds no information. These are the judge's real recorded verdicts, not a simulation.
What the judge wrote
02 · You vs the judge
Pick the better answer. Say how sure you are.
Five real comparisons where every human voter agreed. After each pick you see what the humans chose and what the three judges said, and at the end, who was right more often and who was more honest about it.
03 · Three judges
Three judges, four presentations, one pair of answers.
Every judge saw every comparison four ways: as published, with the order swapped, with both answers padded, and both. The answers never change, so a judge that is reading them should give the same verdict four times. Red marks a verdict that changed.
Read both answers
Across all comparisons
Each judge's confidence here is its order-swap agreement, the one signal all three have. Dots are the estimate, bars its 95% interval (resampling whole questions). kev-8b is split by whether the prompt fits its training length, auto-j by conversation turn.
04 · Ship it?
Auto-accept the confident verdicts. How many wrong ones get through?
A pipeline accepts every verdict whose confidence clears a threshold and sends the rest to a human. Each dot below is one recorded verdict; red ones are wrong.
agreement with humans, chance-corrected
05 · How it was tested
From human votes to honest error bars.
The whole pipeline, one step per box. Select a step to see what happens there.
The full write-up, with every figure, table and confidence interval: REPORT.md.