LLM Judge
Juri LLM
Using a model to score another model's output against a rubric.
An LLM judge is a model prompted to score outputs. You give it the task, the expected answer, the rubric, and the output to grade, and it returns a score with reasoning.
It is how you evaluate open-ended text at a scale humans cannot. It is also where evaluation quietly becomes circular: if you judge with the same family of model that produced the answer, you are partly grading its own style.
A judge needs calibration. Run it against outputs humans have already graded, measure how often it agrees, and check that it agrees on the cases that matter, not just on average. A judge that scores everything 4 out of 5 has told you nothing.
Related terms
Evaluation
Measuring whether the system does the job, on a fixed set of cases, repeatedly.
Benchmark
A shared test that lets you compare systems. Useful, and easy to over-trust.
Rubric
A written set of criteria a judge, human or model, scores against.
Golden Dataset
A fixed, reviewed set of real cases the system is measured against.
Want this working inside your team?
It starts with one conversation about your workflow.