Glossary

LLM as a judge

LLM as a judge is a method where a language model evaluates the output of another model, or its own, against stated criteria and returns a verdict or a score. It is used to check AI work at a volume people cannot review by hand, before the result is accepted.

How LLM as a judge works

The judge gets the task, the result and the acceptance criteria it must meet, and answers whether each one is met and why. A clear criterion such as "every step has an owner" is easy to judge; a vague one such as "good quality" gives vague verdicts.

Where LLM as a judge needs a person

A model judge can share the blind spots of the model it checks, and it can be confidently wrong. It works well for clear, checkable criteria and large volumes; decisions with money, legal or reputational risk still need a human in the loop.

Why check AI output at all

Without a check, "done" means whatever the person or the model says. Checking every result against criteria written in advance makes done mean the same thing for everyone, which is the core of harness engineering, the third of the four stages of AI maturity.