The second step of four. Build a dataset from production traces, score it with judges you write in Python, run the experiment from your own CI, then read the per-case diff. This page is about the scoring, which is the step that decides whether the other three were worth doing.
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import LLMJudge
dataset = Dataset(
cases=[Case(inputs='Can I return this after 40 days?')],
evaluators=[
LLMJudge(rubric='States the 30-day return window.'),
],
)
report = dataset.evaluate_sync(answer_question)The interesting failures in an LLM application have no exact answer to compare against. Whether a reply stayed inside the retrieved context, whether it hedged where it should have refused, whether the tone suited an apology: none of that is a string comparison.
A judge is a model you point at the output with criteria you wrote, called a rubric. It reads the result the way a reviewer would and returns a verdict you can collect across hundreds of cases. Everything that makes one work, or not work, is in the rubric.
Use one only where a cheaper check cannot reach. Structure, schema, required fields, forbidden strings, a number in range: those are deterministic, cost nothing and never drift. A suite that judges everything is slow, expensive and harder to trust than one that judges the three things a regex cannot.
Nearly every judge that behaves oddly is a rubric problem, not a model problem. The rubric is the part you own, and it is the part worth the time.
LLMJudge(
rubric=(
'The answer states the 30-day return window and does not '
'invent an exception to it. Refusing to answer is not a pass.'
),
# The judge cannot check an answer against a question it was
# never shown. Off by default, and the usual reason a rubric
# about relevance scores nonsense.
include_input=True,
model='openai:gpt-5-mini',
assertion={'evaluation_name': 'refund_policy', 'include_reason': True},
)A rubric describing what good looks like leaves the judge to guess where the line is, and it will put the line somewhere different next week. Describe the specific thing that would make this output wrong. 'Invents a policy exception' is checkable. 'Is helpful' is not.
Judges are agreeable. A rubric that only describes success finds success, including in an answer that dodged the question politely. Adding the failure case, such as a refusal not counting as a pass, changes the verdict on exactly the outputs you built the eval to catch.
A rubric asking about grounding and tone and length returns one verdict for three different things, and you cannot tell which one failed. Separate judges cost more and tell you where the problem is, which is the reason you are running this at all.
include_input and include_expected_output are both off by default. A rubric about whether the answer addressed the question cannot work without the question, and this is the usual cause of a judge that scores confidently and wrongly.
A judge can return a pass/fail assertion, a 0 to 1 score, or both from the same call. Gate CI on the assertion, because a build needs a decision. Track the score where a quality moves gradually and you want the trend.
# A verdict, for the suite your CI gates on. This is the default.
LLMJudge(
rubric='States the 30-day return window.',
assertion={'include_reason': True},
)
# A 0-1 score, for a quality you want to watch as a trend.
LLMJudge(
rubric='How completely does the answer cover the policy?',
assertion=False,
score={'include_reason': True},
)
Turn on include_reason either way. The judge's reasoning is stored with the verdict, and it is what tells you whether a failure is real
or the rubric was ambiguous. A score with no reasoning behind it is a number you cannot act on.
A judge nobody has checked against a person is a number, not a measurement. The way to find out is to have reviewers label a sample by hand and compare.
Take a few dozen runs the judge has already scored and have a reviewer give each one a verdict, without seeing what the judge said. The annotation queue keeps the whole trace beside the case, so a reviewer is reading the actual run rather than a text field.
Agreement is the number that tells you whether the judge is measuring what you meant. High agreement means you can trust the judge on the rest of the set. Low agreement means the judge and the person are reading the rubric differently, which is a finding about the rubric.
Most disagreement is ambiguity, not incapability. Read the cases where they differ and the wording that caused it is usually obvious. Reaching for a bigger judge model before rereading the rubric is the expensive way to solve a writing problem.
Reviewer verdicts are stored with the run and are queryable, but they do not fold into experiment or live-eval scores. One reviewer's opinion should never quietly move a number your CI gates on, and keeping them separate is what lets you use them to audit the judge.
One judgment is an anecdote. The value shows up when the same judges run against a new prompt or a new model and you can read what moved, case by case rather than as an average.
Because every judgment is recorded as a span, the aggregates are not the only view. The same results are queryable, so "which judge fails most often, by customer tier" is a query rather than a feature request.
select
attributes->>'gen_ai.evaluation.name' as judge,
count(*) filter (
where attributes->>'gen_ai.evaluation.score.value' = 'true'
) as judge_passed,
count(*) as scored
from records
where attributes->>'gen_ai.evaluation.name' is not null
and start_timestamp > now() - interval '7 days'
group by judge
order by scored desc;A dataset only contains the cases you thought to write down. Running the same judges against real traffic is how you find the inputs it does not contain yet, which is most of them. Offline judging tells you whether to ship; online judging tells you what to add to the dataset next.
You choose where the judge runs. Logfire can run it for you against an agent's traffic, with no evaluator code in your application at all. Or run the judge in your own code and emit the results yourself, which is what you want when the payload should not leave your side of the line.
The rubric is the same either way, and so is where the verdicts end up: hosted and self-run scores land together in Live Evaluations and answer the same SQL.
Scoring is the second step of four: build a dataset, score it, run the experiment, read the diff. A judge is only as useful as the cases you point it at, and the cases that matter most are the ones that already went wrong in production.
The full workflow, including building datasets from real traces and comparing a candidate against a baseline, is on the evals page. If you are choosing a judge model, the traces behind each run are on the LLM observability page.
It is using a language model to score another model's output against criteria you write, called a rubric. It exists because the interesting failures in an LLM application have no exact answer to compare against. You cannot regex your way to whether a reply was grounded in the retrieved context, whether it hedged when it should have refused, or whether the tone suited an apology. A judge reads the output the way a reviewer would, applies your rubric, and returns a verdict you can collect over hundreds of cases.
Whenever a cheaper check would do. Structure, schema, required fields, forbidden strings, valid JSON, a number in range: all of those are deterministic, cost nothing, never drift and never need their own evaluation. Reach for a judge only for the qualities that need reading. A suite that judges everything is slow, expensive, and harder to trust than one that judges the three things a regex cannot reach.
Be specific about the failure you are trying to catch, not about quality in general. 'Good response' gives the judge nothing to apply consistently. 'States the 30-day return window and does not invent an exception to it' can be applied the same way twice. Name what counts as a failure as well as a pass, because judges are agreeable by default and a rubric that only describes success tends to find it.
A pass/fail assertion is what you gate CI on, because a build needs a decision. A 0 to 1 score is better for a quality that moves gradually and that you want to watch as a trend across runs. You can ask for both from one judge. Turn on include_reason either way: the judge's reasoning is what tells you whether a failure is real or the rubric was ambiguous.
Not necessarily the one under test, and not automatically the largest. Match the judge to the difficulty of the rubric: a cheap model is fine for a check that is nearly mechanical, and a stronger one earns its cost where the judgment is subtle. Pin the model and pin temperature to 0, because a judge that gives different verdicts on the same output across runs makes every comparison meaningless.
Only if you ask it to. include_input passes the original input and include_expected_output passes the reference, and both are off by default. This is the most common reason a rubric behaves strangely: a rubric about relevance cannot work if the judge was never shown what the user asked. Turn them on deliberately, since a judge shown the expected answer will tend to reward matching it rather than being right.
Measure it against people. Have reviewers label a sample of the same runs by hand, then compare their verdicts with the judge's. Where they disagree you have learned something: usually that the rubric is ambiguous, occasionally that the judge is not strong enough for it. Until you have done that once, you are trusting the judge on its own word.
It can, and it does not have to. Logfire can run the judge against an agent's traffic, so there is no evaluator code in your application. Running it yourself is the other option, and the one you want when the payload should not leave your side of the line, or when what you are judging is not an agent run. The rubric is the same either way.
Evaluation results use Logfire's normal telemetry allowance instead of a separate per-score meter. Personal includes 10 million records per month and pauses ingestion at the limit; Team and Growth charge $2 per million additional records after the same allowance. The judge's own model calls are billed by whichever provider you point it at.
Get started with 10 million free spans, logs, and metrics per month. No credit card required.