LLM-as-judge became standard technique by late 2024 and remains, in 2026, the only scalable way to evaluate qualitative quality in LLM systems. The question distinguishing mature teams isn’t whether to use it, but when to trust the number it produces.

Key takeaways

  • Dimensional rubric (6 dimensions → 1-5 score + justification) produces more stability than a global score.

  • Judge-human correlation must exceed 0.7 on a 30-case sample to consider the judge usable.

  • The judge systematically fails on subjective criteria without rubric, exact ground-truth comparison, and self-evaluation.

  • Full calibration quarterly or when changing model; quick 5-case check on any anomaly.

  • If the criterion can be checked with deterministic logic, don’t use an LLM judge.

What an LLM judge does well

Dimensional rubric evaluation: coherence, relevance, format, absence of fabricated info, style adherence. Asking the judge for 1-5 scores on each dimension with one-sentence justification produces results correlating reasonably with humans when well calibrated.

The study that popularized the technique is Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena[1] (Zheng et al., 2023). It measured that strong judges like GPT-4 reach over 80% agreement with human preferences, the same level of agreement found between two human evaluators. Anthropic’s evaluation guide[2] recommends the same pattern in production: detailed rubrics and asking the model to reason before scoring, because it improves performance on complex judgments.

The key is decomposition:

  • Asking for a "global score" produces high inconsistency between runs.

  • Asking for six dimensional scores and aggregating later produces stability. The G-Eval[3] paper (Liu et al., 2023) confirms the pattern with data. Chain-of-thought reasoning plus per-dimension scores aggregated afterward reaches a Spearman correlation of 0.514 with human evaluators on summarization tasks, well above prior reference metrics.

What an LLM judge does poorly

Three patterns where it consistently fails:

  • Subjective criteria without rubric: "is this answer useful?" without defining what counts as useful produces noise, not signal.

  • Comparison against exact ground truth: the judge accepts correct paraphrases but also incorrect ones that sound similar.

  • Evaluation when the judge is the same model being evaluated: systematically overrates its own family.

  • Position bias: the order in which responses appear skews the score. Wang et al. (2023)[4] showed that, with ChatGPT as judge, a weaker response (Vicuna-13B) could "beat" ChatGPT in 66 of 80 comparisons by flipping the presentation order.

The same problem shows up when evaluating retrieval: as explained in how to evaluate a RAG system without fooling yourself, a judge without a reliable human reference amplifies noise instead of reducing it.

Human calibration

The verification you can’t work without is calibrating the judge against humans on a small subset:

  • Humans score 30 cases.

  • The judge scores the same 30.

  • The correlation between them should be above 0.7 to consider it usable.

  • Below 0.7: adjust the judge’s prompt or change model.

Recommended frequency:

  • Full calibration quarterly, or when changing the judge model.

  • Quick check (~5 cases) whenever anomalous behaviour appears in metrics.

This is the same calibration loop we describe in production-grade agent evaluations: without a human reference sample, any automated metric ends up measuring noise, not quality.

When NOT to use LLM judge

When verification can be automated with deterministic logic:

  • Valid schema.

  • Resolvable URL.

  • Correct type.

  • Value within expected range.

Use that logic: cheaper, more reliable, no calibration. LLM judge only when the criterion is genuinely qualitative.

Conclusion

LLM-as-judge in 2026 is a mature technique with known limits. Used with dimensional rubric, periodic calibration, and as complement (not substitute) to deterministic metrics, it produces useful signal for CI regression and drift detection. Used with blind trust, it’s a mirror reflecting the judge’s biases without anyone noticing.

This article is also available in Spanish.

Sources:

  1. Zheng et al. — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena[1]
  2. Liu et al. — G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment[3]
  3. Wang et al. — Large Language Models are not Fair Evaluators[4]
  4. Anthropic — Define success criteria and build evaluations[2]

Frequently asked questions

How do I check whether my LLM judge is reliable before using it in CI?

By calibrating it against humans on a small subset: humans score 30 cases, the judge scores the same 30, and the correlation between them must exceed 0.7 to consider it usable. If it falls below, adjust the judge's prompt or change the model. Repeat full calibration quarterly or whenever you change the judge model, and run a quick check of about 5 cases whenever the metrics show anomalous behaviour.

Why ask for six per-dimension scores instead of one global score?

Because a global score produces high inconsistency between runs, whereas six dimensional scores (coherence, relevance, format, absence of fabricated information, style adherence) with a one-sentence justification, aggregated afterwards, produce stability. G-Eval (Liu et al., 2023) confirms it: chain-of-thought reasoning plus per-dimension scores reaches a Spearman correlation of 0.514 with human evaluators on summarization tasks, well above prior reference metrics.

Can I use the same model as judge of its own answers?

Not advisable: the judge systematically overrates its own model family. It is not neutral to order either: Wang et al. (2023) measured this with ChatGPT as judge. A weaker Vicuna-13B response could beat ChatGPT in 66 of 80 comparisons by flipping the presentation order. And if the criterion can be checked with deterministic logic (valid schema, resolvable URL, correct type, value within range), do not use an LLM judge at all.

Sources

  1. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
  2. Anthropic’s evaluation guide
  3. G-Eval
  4. Wang et al. (2023)