LLM-as-Judge Reliability: How to Measure It

Using an LLM to score another LLM's output — or a human's — is common enough now that it needs its own reliability bar, separate from "the underlying model is good." A judge model can be capable and still be an unreliable judge, in the specific sense that its scores don't correspond consistently to what a domain expert would say. The only way to know the difference is to check the judge against something you trust — and that's exactly the piece that's easiest to build last, or not at all.

A concrete failure mode: a measurement system with no measurements

It's possible to build every visible piece of a calibration system — a service that compares a judge's score to an expected score, a deviation log, a threshold-based warning, a periodic worker that runs the comparison, even an admin dashboard summarizing the results — and still have it produce zero real comparisons, indefinitely, because nothing ever populates the ground-truth data the comparison logic depends on. The comparison function itself may be correct. The dashboard may render cleanly. The system can look complete from every angle except the one that matters: it has never actually checked anything.

This is a specific, checkable failure, not a hypothetical one — and it's worth checking for directly, not inferring from the presence of calibration-shaped code. The check is simple: does anything, anywhere, write a row into the ground-truth table this system reads from? If the answer is no, the calibration system is a wiring diagram, not a working measurement.

What ground truth actually requires

Ground truth for a judge model means: expert-scored examples the judge's output can be compared against, ideally spanning the range of quality the judge will encounter in production — not just clearly-good and clearly-bad cases, but the ambiguous middle where judges most often disagree with experts and with each other. Populating this is deliberate, ongoing work: someone has to score examples, on a cadence, and that cadence has to be treated as part of the system's maintenance, not a one-time setup task.

The trade-off worth stating plainly

Building the calibration scaffolding — the comparison logic, the logging, the dashboard — before you have any ground truth to check it against isn't wrong. It's a reasonable order of operations if the ground-truth population is scheduled as the very next step. It becomes a problem specifically when the scaffolding is mistaken for the finished system, because everything about it looks finished: tests can pass against synthetic inputs, the code compiles, the dashboard renders. None of that tells you whether the judge is actually reliable.

A minimum bar before calling a judge system "reliable"

At minimum: a non-empty, periodically refreshed set of expert-scored examples; a defined deviation threshold with a documented reason for that specific number, not a default left unexamined; a review cadence for cases that exceed it — not just a log entry no one reads; and — the part easiest to skip — someone accountable for actually looking at the calibration results on a schedule, the same way the routing-weight review matters for a cost-aware orchestration layer. A calibration system with no one assigned to review its output has the same failure mode as one with no ground truth at all: it exists, it runs, and it changes nothing.

The practical recommendation

Before describing an LLM-as-judge system as reliable, or as calibrated, check for ground truth directly rather than assuming it exists because the surrounding code does. If it doesn't exist yet, say so explicitly, schedule its creation as real work with an owner, and treat the comparison scaffolding as incomplete infrastructure until it does — not as a finished reliability guarantee.