Human Evaluation & LLM-as-Judge Calibration

Human Evaluation and LLM Judge Calibration

Most interesting LLM outputs cannot be checked with exact-match tests: a support answer can be correct in many wordings, a summary can be accurate but miss the key point, and a refusal can be appropriate or overcautious. Teams therefore rely on human evaluation and increasingly on LLM-as-a-judge — using a model to grade outputs at scale. Both are only as useful as they are consistent and aligned with what users and the business care about.

An uncalibrated judge is worse than none: it produces confident numbers that drift away from reality, rewards verbose answers, and quietly approves the failures you most needed to catch. This page covers building rubrics people can apply consistently, measuring agreement between people, and calibrating model judges against human labels before trusting them in regression gates and dashboards.

TL;DR

Write rubrics with binary, specific criteria and worked examples; avoid vague 1–10 scales.
Have several people label a sample; measure inter-annotator agreement (Cohen's or Krippendorff's kappa/alpha) and adjudicate disagreements into a gold set.
Use LLM judges per criterion, in pointwise (pass/fail) or pairwise (A vs B) mode, with reasoning before the verdict.
Calibrate: compare judge labels with the gold set — report true-positive and true-negative rates and kappa, not only accuracy.
Correct known biases: position (swap order), verbosity, self-preference, and deference to confident claims.
Re-check judges when prompts, models, or data change, and keep evaluation sets covering known failure modes.

Quick Example

Measuring how well a judge agrees with adjudicated human labels on a pass/fail criterion:

A pairwise judgment that controls for position bias by asking twice with the order swapped:

A criterion-specific judge prompt:

Core Concepts

Rubrics That People Can Apply

Binary criteria are easier to apply consistently and easier to turn into actionable metrics. Derive criteria from error analysis of real outputs — read failures first, then name them — rather than from a generic list.

Human Labeling and Agreement

Have two or more trained annotators label the same sample. Inter-annotator agreement shows whether the rubric is clear: Cohen's kappa for two raters, Krippendorff's alpha for more raters or missing labels. Low agreement means the criterion or guidelines need work, not that annotators are careless. Adjudicate disagreements — a lead reviewer decides and updates the guidelines — and the adjudicated labels become the gold set for calibrating judges and regression testing. Domain experts are needed for medical, legal, financial, or technical correctness.

LLM Judges

Good practice: one criterion per judge call, explicit definitions, few-shot examples from the gold set, brief reasoning before the verdict, and constrained output for parsing. See Prompt Engineering.

Calibration

Accuracy alone hides problems when most outputs pass: a judge that always says "pass" can score 90% accuracy while catching no failures. Report:

True-positive rate: of answers humans marked good, how many the judge passed.
True-negative rate: of answers humans marked bad, how many the judge caught — usually the number that matters most.
Cohen's kappa: agreement corrected for chance.

Iterate on the judge prompt using a development split of the gold set and report final numbers on a held-out split, so you do not overfit the judge to its own test.

Known Biases

Operating Evaluations

Coverage: maintain examples for each known failure mode — hallucinated facts, unsafe advice, missed escalation, wrong language — and add new ones from production incidents. See LLM Evaluation Datasets and Regression Gates.
Drift: re-run calibration whenever the judge model, judge prompt, application prompt, or user population changes.
Sampling production: periodically send a stratified sample of live outputs to human review, and compare with the judge.
Cost and latency: use smaller judges for high-volume monitoring once calibrated, and stronger judges or humans for disputed cases.
Online signals: combine offline judgments with user feedback and A/B tests. See A/B Testing.

Best Practices

Start With Error Analysis

Read and categorize real failures before writing rubrics or judges; criteria should describe failures you actually see.

Keep Humans in the Loop

Use LLM judges to scale human judgment, not replace it; refresh gold labels regularly.

Report Judge Quality With Every Metric

Show the judge's TPR, TNR, and kappa alongside scores so readers know how much to trust them.

Version Everything

Version rubrics, judge prompts, judge models, and gold sets, so score changes can be traced to causes.

Common Mistakes

Vague Numeric Scales

Trusting an Uncalibrated Judge

Calibrating on the Same Examples Used to Tune the Judge

FAQ

What is LLM-as-a-judge?

Using a language model to evaluate outputs of an LLM application against criteria — such as correctness, groundedness, or tone — so evaluation can run at a scale and speed humans cannot match.

How do I know if an LLM judge is reliable?

Compare its labels with adjudicated human labels on a held-out set and report true-positive rate, true-negative rate, and Cohen's kappa. Re-check whenever the judge or application changes.

Should I use scores or pass/fail?

Pass/fail per specific criterion is usually more consistent and actionable than numeric scales. Pairwise comparisons work well for choosing between versions.

What is inter-annotator agreement?

A measure of how consistently different people apply the same labels, corrected for chance agreement. Low agreement indicates an unclear rubric or a genuinely subjective criterion.

How many human labels do I need?

Enough to estimate judge agreement with useful confidence — often a few hundred examples per important criterion, including enough failures to measure the true-negative rate.