Score answers against criteria you can read. Every verdict keeps the judge's reason and links to the trace that produced it, so you can tell whether the response failed or the rubric did.
Trusted by teams building production software and AI
The verdict is an index into evidence, not the end of the investigation. Open the case to see the input, the answer, what the answer was expected to do, and the judge's reason in one place.
The trace link continues into the model and tool calls that produced the answer. That is where you can tell whether the response broke the rubric, the system skipped a required tool, or the rubric asked the wrong question.
A judge nobody has checked against a person is a number, not a measurement. Have reviewers label a sample by hand, then compare the two verdicts.
Take a few dozen runs the judge has already scored and have a reviewer give each one a verdict, without seeing what the judge said. Save the verdict, expected output, and review notes on that same run so the comparison has a stable case ID rather than a copied text field.
Join the automated result to the reviewer annotation on the same run. High agreement means the rubric is measuring what you meant. Low agreement means the judge and the person are reading it differently, which is a finding about the rubric.
Most disagreement is ambiguity, not incapability. Read the cases where they differ and the wording that caused it is usually obvious. Reaching for a bigger judge model before rereading the rubric is the expensive way to solve a writing problem.
Reviewer verdicts are stored with the run and are queryable, but they do not fold into experiment or live-eval scores. One reviewer's opinion should never quietly move a number your CI gates on, and keeping them separate is what lets you use them to audit the judge.
The interesting failures in an LLM application have no exact answer to compare against. Whether a reply stayed inside the retrieved context, whether it hedged where it should have refused, whether the tone suited an apology: none of that is a string comparison.
A judge is a model you point at the output with criteria you wrote, called a rubric. It reads the result the way a reviewer would and returns a verdict you can collect across hundreds of cases. Everything that makes one work, or not work, is in the rubric.
Use one only where a cheaper check cannot reach. Structure, schema, required fields, forbidden strings, a number in range: those are deterministic, cost nothing and never drift. A suite that judges everything is slow, expensive and harder to trust than one that judges the three things a regex cannot.
Return a pass/fail assertion when a build needs a decision. Return a 0 to 1 score when quality moves gradually and the trend matters. The same judge call can record both, and both should keep the reason that made the result actionable.
# A verdict, for the suite your CI gates on. This is the default.
LLMJudge(
rubric='States the 30-day return window.',
assertion={'include_reason': True},
)
# A 0-1 score, for a quality you want to watch as a trend.
LLMJudge(
rubric='How completely does the answer cover the policy?',
assertion=False,
score={'include_reason': True},
)// A verdict, for the suite your CI gates on. This is the default.
new LLMJudge({
rubric: 'States the 30-day return window.',
assertion: { evaluationName: 'judge_pass' },
})
// A 0-1 score, for a quality you want to watch as a trend.
new LLMJudge({
rubric: 'How completely does the answer cover the policy?',
score: { evaluationName: 'judge_score' },
})
// There is no includeReason option here as there is in Python.
// The reasoning is whatever your judge callback returns as
// `reason`, and it is stored with the verdict either way.
setDefaultJudge(async ({ output, rubric }) => ({
pass: await grade(output, rubric),
reason: 'Cites the 30-day window without inventing a carve-out.',
}))Nearly every judge that behaves oddly is a rubric problem, not a model problem. The rubric is the part you own, and it is the part worth the time.
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import LLMJudge
dataset = Dataset(
cases=[Case(inputs='Can I return this after 40 days?')],
evaluators=[
LLMJudge(
rubric=(
'The answer states the 30-day return window and does not '
'invent an exception to it. Refusing to answer is not a pass.'
),
# Relevance needs the question as well as the answer.
include_input=True,
model='openai:gpt-5-mini',
assertion={
'evaluation_name': 'refund_policy',
'include_reason': True,
},
),
],
)
report = dataset.evaluate_sync(answer_question)import {
Case, Dataset, LLMJudge, setDefaultJudge,
} from 'logfire/evals'
// TypeScript ships no model client: name the judge once.
setDefaultJudge(async ({ output, rubric }) =>
askYourModel(output, rubric)
)
const dataset = new Dataset({
cases: [new Case({ inputs: 'Can I return this after 40 days?' })],
evaluators: [
new LLMJudge({
rubric:
'The answer states the 30-day return window and does not ' +
'invent an exception to it. Refusing to answer is not a pass.',
// Relevance needs the question as well as the answer.
includeInput: true,
assertion: { evaluationName: 'refund_policy' },
}),
],
})
const report = await dataset.evaluate(answerQuestion)A rubric describing what good looks like leaves the judge to guess where the line is, and it will put the line somewhere different next week. Describe the specific thing that would make this output wrong. 'Invents a policy exception' is checkable. 'Is helpful' is not.
Judges are agreeable. A rubric that only describes success finds success, including in an answer that dodged the question politely. Adding the failure case, such as a refusal not counting as a pass, changes the verdict on exactly the outputs you built the eval to catch.
A rubric asking about grounding and tone and length returns one verdict for three different things, and you cannot tell which one failed. Separate judges cost more and tell you where the problem is, which is the reason you are running this at all.
include_input and include_expected_output are both off by default. A rubric about whether the answer addressed the question cannot work without the question, and this is the usual cause of a judge that scores confidently and wrongly.
Start free with 10 million spans, logs, and metrics each month. No credit card required.
One judgment is an anecdote. The value shows up when the same judges run against a new prompt or a new model and you can read what moved, case by case rather than as an average.
Because every judgment is recorded as a span, the aggregates are not the only view. The same results are queryable, so "which judge fails most often, by customer tier" is a query rather than a feature request.
select
attributes->>'gen_ai.evaluation.name' as judge,
avg(cast(attributes->>'gen_ai.evaluation.score.value' as double))
as pass_rate,
count(*) as scored
from records
where attributes->>'gen_ai.evaluation.score.label'
in ('pass', 'fail')
and start_timestamp > now() - interval '7 days'
group by judge
order by scored desc;A dataset only contains the cases you thought to write down. Running the same judges against real traffic is how you find the inputs it does not contain yet, which is most of them. Offline judging tells you whether to ship; online judging tells you what to add to the dataset next.
You choose where the judge runs. Logfire can run it for you against an agent's traffic, with no evaluator code in your application at all. Or run the judge in your own code and emit the results yourself, which is what you want when the payload should not leave your side of the line.
The rubric is the same either way, and so is where the verdicts end up: hosted and self-run scores land together in Live Evaluations and answer the same SQL.
Scoring is the second step of four: build a dataset, score it, run the experiment, read the diff. A judge is only as useful as the cases you point it at, and the cases that matter most are the ones that already went wrong in production.
The full workflow, including building datasets from real traces and comparing a candidate against a baseline, is on the evals page. If you are choosing a judge model, the traces behind each run are on the LLM observability page.
It is using a language model to score another model's output against criteria you write, called a rubric. It exists because the interesting failures in an LLM application have no exact answer to compare against. You cannot regex your way to whether a reply was grounded in the retrieved context, whether it hedged when it should have refused, or whether the tone suited an apology. A judge reads the output the way a reviewer would, applies your rubric, and returns a verdict you can collect over hundreds of cases.
Whenever a cheaper check would do. Structure, schema, required fields, forbidden strings, valid JSON, a number in range: all of those are deterministic, cost nothing, never drift and never need their own evaluation. Reach for a judge only for the qualities that need reading. A suite that judges everything is slow, expensive, and harder to trust than one that judges the three things a regex cannot reach.
Be specific about the failure you are trying to catch, not about quality in general. 'Good response' gives the judge nothing to apply consistently. 'States the 30-day return window and does not invent an exception to it' can be applied the same way twice. Name what counts as a failure as well as a pass, because judges are agreeable by default and a rubric that only describes success tends to find it.
A pass/fail assertion is what you gate CI on, because a build needs a decision. A 0 to 1 score is better for a quality that moves gradually and that you want to watch as a trend across runs. You can ask for both from one judge. Turn on include_reason either way: the judge's reasoning is what tells you whether a failure is real or the rubric was ambiguous.
Not necessarily the one under test, and not automatically the largest. Match the judge to the difficulty of the rubric: a cheap model is fine for a check that is nearly mechanical, and a stronger one earns its cost where the judgment is subtle. Pin the model and pin temperature to 0, because a judge that gives different verdicts on the same output across runs makes every comparison meaningless.
Only if you ask it to. include_input passes the original input and include_expected_output passes the reference, and both are off by default. This is the most common reason a rubric behaves strangely: a rubric about relevance cannot work if the judge was never shown what the user asked. Turn them on deliberately, since a judge shown the expected answer will tend to reward matching it rather than being right.
Measure it against people. Have reviewers label a sample of the same runs by hand, then compare their verdicts with the judge's. Where they disagree you have learned something: usually that the rubric is ambiguous, occasionally that the judge is not strong enough for it. Until you have done that once, you are trusting the judge on its own word.
It can, and it does not have to. Logfire can run the judge against an agent's traffic, so there is no evaluator code in your application. Running it yourself is the other option, and the one you want when the payload should not leave your side of the line, or when what you are judging is not an agent run. The rubric is the same either way.
Evaluation results use Logfire's normal telemetry allowance instead of a separate per-score meter. Personal includes 10 million records per month and pauses ingestion at the limit; Team and Growth charge $2 per million additional records after the same allowance. The judge's own model calls are billed by whichever provider you point it at.
Get started with 10 million free spans, logs, and metrics per month. No credit card required.