Build datasets from production traces, define scorers in Python, compare a candidate against a baseline case by case, and review by hand where it matters. Evaluation results are queryable telemetry records covered by the normal plan allowance, not a separate per-score meter.
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import LLMJudge
dataset = Dataset(
cases=[Case(inputs='Return after 40 days?')],
evaluators=[LLMJudge(rubric='States the 30-day window.')],
)
report = dataset.evaluate_sync(answer_question)An eval is a test for behavior that has no single correct answer. Rather than asserting one exact string, you collect a dataset of cases, define scorers that judge each result, and watch how the scores move as you change the prompt, the model or the retrieval.
The number itself is not the point. The point is being able to say this change made things better, and to say it before your users find out otherwise.
Offline evaluation runs against a curated dataset while you are developing, so it catches regressions before you deploy. It is the closest thing an LLM application has to unit tests, and it belongs in CI for the same reason.
Online evaluation scores real production traffic after the fact. It is the only way to see the inputs your dataset does not contain yet, which is most of them. Offline tells you whether to ship; online tells you what to add to the dataset next.
Cases come from four places: production traces you found going wrong, user feedback, hand-written examples of behavior you care about, and synthetic variations of all three. The first is the one that compounds: a run you saw fail in Live view can be saved directly as a case, so the bug report and the regression test are the same object rather than two things you keep in sync.
A scorer decides whether one result was good. Deterministic checks handle structure, schema and forbidden content, and cost nothing. LLM judges handle the qualitative questions a regex cannot express, with a rubric you write. Human review covers what you do not yet trust either to decide. Real suites use all three, because each is wrong in a different way.
You run the evaluation, not us. Call it from CI, from a script, or from your machine, with your model credentials and your data; Logfire records the experiment and stores every case-level result. That boundary is deliberate: your evals stay in your pipeline, and what you get here is the history, the comparison and the query surface over it.
Pick a baseline experiment and a candidate, and read the delta per scorer and per case. The summary tells you what moved; the case view tells you which inputs moved it, which is the part that decides whether to ship. Direction is something you set per scorer, because a higher number is not automatically an improvement: latency and cost go the other way.
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import IsInstance, LLMJudge
dataset = Dataset(
cases=[
Case(
name='refund_policy',
inputs='Can I return this after 40 days?',
expected_output='No, the return window is 30 days.',
),
],
evaluators=[
IsInstance(type_name='str'),
LLMJudge(rubric='States the 30-day window, invents no exception.'),
],
)
report = dataset.evaluate_sync(answer_question) pydantic-evals works with any Python you can call, not only Pydantic AI, and a custom scorer is an ordinary function. Because the suite
is code, it is reviewed in pull requests and runs in CI beside your other tests.
Note what this means: Logfire does not run your evals. You run them, with your credentials and your data, and Logfire records the experiment, stores every case-level result, and gives you the history and the comparison over it.
pydantic-evals gives you Case, Dataset and a library of evaluators, and a custom scorer is an ordinary function. That means your evals live in your repository, get reviewed in pull requests, and run in CI beside your other tests. On most eval products the scorer is configured in the vendor's UI instead, which is a different thing wearing the same word: it cannot be code-reviewed, and it does not travel with the branch that changed the prompt.
An aggregate score that moved two points does not tell you whether you fixed three cases and broke one. The comparison view lists every case with its baseline result, its candidate result and the outcome, so you can read the cases that changed rather than trusting the mean, which is the difference between knowing a change is safe and hoping it is.
Reviewers work a queue and record a verdict, a category, an expected output and tags against a run, with the whole trace visible beside it. Verdicts are stored with the run and are queryable, so they drive triage and dataset curation. They are deliberately kept out of experiment and live-eval aggregates, so one reviewer's opinion never quietly moves a number your CI gates on.
Your application emits evaluation results against real traffic and Logfire reads them, so a scorer can watch live requests rather than only the dataset you thought to write. Note where the work happens: we do not reach into your traffic and score it for you, which some products do. You decide what gets scored and how often, in your code, and that is also why scoring production never becomes a bill you did not choose.
Keep prompts out of your code as versioned, labeled templates the SDK fetches at runtime, roll one out to a percentage of traffic, and roll it back without shipping. Every resolution is recorded, so an experiment result can be traced back to the exact prompt version that produced it.
Evaluation results are stored as ordinary queryable telemetry records rather than behind a separate score API. They use the same plan allowance instead of a per-score meter, and the same SQL can answer 'which scorer regressed most this week, by customer tier' without waiting for a purpose-built dashboard.
Live evals read the evaluation results your application emits against real traffic, so a scorer can watch production rather than only the dataset you thought to write. Because those results are OpenTelemetry events in the same records table as your traces, watching them is a query:
Worth being exact about the boundary, because it is the same one as the Run button. Some products score your production traffic for you, server-side, and that is genuinely less work. Here the scoring happens in your application, which means the prompt, the response and the judge's reasoning never have to leave your side of the line for a score to exist, and the volume you score is a decision you make rather than one that arrives on an invoice.
select
attributes->>'gen_ai.evaluation.name' as scorer,
avg(cast(attributes->>'gen_ai.evaluation.score.value' as double))
as mean_score,
count(*) as scored_runs
from records
where attributes->>'gen_ai.evaluation.score.value' is not null
and start_timestamp > now() - interval '24 hours'
group by scorer
order by mean_score asc;
Sampling is configured in your code, so how much of production gets scored stays your decision. And because it is SQL, the same question broken
down by customer, by model or by prompt version is the same query with one more group by.
Some failures no scorer catches: the answer was right but the tone was wrong, or it guessed where it should have looked something up. Those get marked by a person, on the run itself, and the verdict is stored as structured data rather than as a note in someone's head.
In a month, we were already 10 times better, just because we could really control the state of each of the steps that we have.
An eval is a test for behavior that has no single correct answer. Instead of asserting one exact output, you collect a dataset of cases, define scorers that judge each result, and track how the scores move as you change the prompt, the model or the retrieval. The point is not a pass or fail number, it is being able to tell whether a change made things better or worse before it reaches users.
Offline evaluation runs against a curated dataset while you are developing, so it catches regressions before you deploy. It is the closest thing an LLM application has to unit tests. Online evaluation scores real production traffic after the fact, which is the only way to see the inputs your dataset does not contain yet. Most teams need both: offline to gate a change, online to find the cases the dataset is missing.
No, and this is worth being precise about. You run an evaluation from your own code, with pydantic-evals or anything else that emits the OpenTelemetry GenAI evaluation events. Logfire records the experiment, stores it, and lets you compare it against a baseline case by case. There is no Run button in the UI, which means your evals run in your CI, on your machine, with your data and your model credentials.
When you find a run that went wrong in the Live view, you can save that span directly as a dataset case rather than retyping the input somewhere else. That is the loop the whole category talks about and it is the reason evals and tracing belong in one product: the bug report and the regression test are the same object.
Yes, and most useful eval suites do all three. Deterministic checks are cheap and exact, so use them for structure, schema and forbidden content. LLM judges handle the qualitative questions a regex cannot. Human review is for the cases where you do not yet trust either, and for building the labeled set that tells you whether your judge agrees with you.
Reviewers work through a queue and record a verdict, a category, an expected output and tags against a run. Those verdicts are stored with the run and are queryable like anything else, so they are useful for triage and for curating a dataset. They are deliberately not folded into experiment or live-eval aggregates, so a reviewer's opinion never silently moves a score your CI is gating on.
Evaluation results use Logfire's normal telemetry allowance rather than a separate per-score meter. Personal includes 10 million records per month and pauses ingestion at the limit. Team and Growth include the same allowance, then charge $2 per million additional records. That distinction matters once a nightly suite runs thousands of scores a night.
No. pydantic-evals works with any Python code you can call, including applications built on other frameworks or on raw provider SDKs, and Logfire ingests evaluation results from anything emitting the OpenTelemetry GenAI evaluation conventions. Using Pydantic AI means the traces underneath are richer, but it is not a requirement for the evals workflow.
Get started with 10 million free spans, logs, and metrics per month. No credit card required.