Can You Trust an LLM to Judge Another LLM?

What is an LLM judge actually good for?

An LLM judge is useful for repeated, rubric-based comparisons where human review is too slow or expensive. It can approximate human preferences, but it does not create ground truth. The judge is another probabilistic model, so its agreement, stability, conflicts, and failures need measurement before its score can guide a release.

Zheng and colleagues' peer-reviewed NeurIPS 2023 paper compared strong LLM judges with controlled and crowdsourced human preferences. GPT-4 reached more than 80 percent agreement, which the authors report as the same level of agreement observed between humans. Their released evidence included 3,000 expert votes and 30,000 conversations with human preferences. For open-ended outputs that lack one exact answer, that scale is difficult to reproduce with expert review alone.

That result supports approximation, not authority. A judge score describes how one configured model applied one rubric to one sample. It is not an objective property of the output. Nazr uses LLM judges when plan quality depends on grounding, focus, or coherence across branches, then treats the judge as an instrument that also needs evaluation. This agentic organization series carries the instrument's evidence grade alongside the result.

What happened when the generator family joined the panel?

Nazr recorded a generator at 4.05/5 when its own model family sat on the judge panel and 3.72/5 under an excluding panel, a 0.33 shift. The first configuration suggested a model upgrade; the second did not. Changing the measuring instrument changed the product decision.

The July 2026 internal report tested whether Nazr should replace its root-priority generator. Under a panel that included the candidate model's family, the candidate appeared to win at 4.05/5. The recorded score fell to 3.72 when an Opus and Gemini panel judged it. The incumbent generator's separately recorded score was 3.88, so the apparent upgrade disappeared.

These are documented internal benchmark results, not an independently reproducible public dataset. The generating harness and saved-output rejudge code remain in the repository, but the raw cells behind this later comparison were not recovered. The result supports a change in evaluation procedure and records why Nazr did not switch the generator. It does not establish a universal 0.33 bias.

Does a 0.33 shift prove self-preference?

No. The 0.33 difference shows that panel composition mattered in this internal test, not how much every model favors its own family. The raw ledger is missing, and a score gap can mix genuine response quality with bias. The safe conclusion is procedural: changing the judge changed the decision.

Chen and colleagues make the confound explicit in their peer-reviewed EMNLP 2025 paper. Self-preference means models “tend to favor their own responses over those generated by other models.” Yet subtracting one model's score from another does not isolate that bias. The judge may genuinely produce stronger answers, so the paper introduces gold judgments as a proxy for actual response quality before estimating self-preference.

Nazr did not run that gold-judgment design for the 0.33 observation. The accurate label is therefore a panel-associated shift consistent with self-preference research. It is the same source discipline used when this publication labels company-reported revenue milestones, separates management announcements from Census evidence, and compares announced AI layoffs with measured hiring changes. The evidence grade travels with the number.

Which biases can change an LLM judge score?

An LLM judge can prefer a response because of its position, length, style, or model family, and can produce a different verdict on repetition. Those effects vary across tasks. A panel can also appear complete after one judge fails unless the harness makes every missing verdict a gate failure.

The NeurIPS study that reported more than 80 percent human agreement also documented position, verbosity, and self-enhancement biases, plus limited reasoning ability. High average agreement and consequential local errors can coexist. A judge can be useful across a benchmark and still reverse the ordering of two close candidates on the slice that determines a release.

Position bias is large enough to measure directly. Shi and colleagues' peer-reviewed IJCNLP-AACL 2025 study tested 15 judges across 22 tasks, about 40 generating models, and more than 150,000 evaluations. The authors found that position bias was not random chance and varied substantially by judge and task. Swapping response order is therefore a test, not a universal cure.

Nazr's code audit found a quieter failure mode. The main coherence panel intends to average two judges, but its aggregator filters failed verdicts and can return a composite from one successful judge. Without the raw run ledger, Nazr cannot claim that every historical two-judge score contains two completed judgments. Panel size belongs in the result, alongside sample size and judge identity.

What controls make an LLM judge trustworthy?

A useful judge pipeline blinds model identity, swaps response order for pairwise tests, validates against human or deterministic labels, repeats noisy comparisons, and reports every judge failure. Several judges help only when their errors differ and all required verdicts arrive. The release gate should fail when its measurement contract is incomplete.

Nazr had several of these controls as defaults and documented rules, but not all as enforced invariants. Judge-family exclusion can still be bypassed by a command-line configuration, and the panel aggregator can accept one verdict. The publication-safe statement is that named runs used generator-excluding panels. Calling the rule structural must wait for the code to enforce it.

The same provenance principle applies to the product. Agent reports and evidence entering Nazr retain their source; evaluation results need an equally inspectable trail of outputs, judge configurations, failures, and decisions.

When should code or humans replace the judge?

Use code when the desired answer can be stated exactly, use humans to establish or audit subjective ground truth, and use an LLM judge for scalable comparisons inside those anchors. Consequential product claims should not rest on one model-scored composite. The measurement method should match the decision the score will trigger.

Nazr's reconciliation benchmark illustrates the code boundary. For a narrow dependency task, fixtures define required and allowed edges, and ordinary code computes precision, recall, and the expected graph outcome. No LLM grades those edges. Humans still authored the answer key, so deterministic scoring makes the result repeatable rather than automatically objective.

Human review belongs at the other boundary: defining whether the rubric captures useful plan quality, validating judge agreement, and examining disagreements that would change a release. Nazr's historical panel had only one documented informal human anchor and no formal inter-rater study for the full rubric. The internal scores can guide development; they cannot prove customer value or general product quality.

Nazr can trust an LLM judge only for the task and decision its calibration covers. Before the next run, that means enforcing generator-family exclusion, requiring every configured judge to complete, and archiving the raw ledger before publishing another exact panel effect. Those controls sit inside the broader planning-system evaluation method, while the production benchmark audit shows what happens when the measurement contract drifts.