Skip to main content

Evaluate

Agent evals that score inside the trace

Evaluate agents and LLM apps on datasets from production traces, run the suite from your code, and compare candidate against a baseline.

Start evaluating free Start free Read the docs
Aggregate comparison of two experiment runs, baseline prompt-v1 against candidate prompt-v2. Summary cards read 17 of 20 to 20 of 20 completed cases, assertions steady at 85 percent, task errors 3 down to 0, and average task duration 765 up to 810 milliseconds. An evaluator analysis table lists safe as an assertion at 75 up to 90 percent, route as a label breakdown that is marked not comparable, and groundedness and quality as scores rising from 0.63 to 0.79 and 0.68 to 0.84, each with a distribution histogram. Below, an operational metrics table marks task duration, input tokens and output tokens all worse, input tokens by 334 percent.

Why evaluation

You can't unit test a language model

Run the same input twice and you can get two different outputs, both arguably correct. “It seems better” ends up being the release decision.

An eval replaces that with a fixed set of cases, scored by code, another model, or a person, run before you ship and against live traffic after. The score lands on the span it scored, next to the tool call and the retry that explain it. The eval is part of the trace, not a separate tool beside it.

How it works

The same scorer, before release and in production

  1. Dataset

    Inputs you control, and the output you expect

  2. Task

    The thing under test, a single call or a whole agent

  3. Scorer

    Code or an LLM judge, run over every case

  4. Experiment

    One run over the dataset, compared against the baseline

  5. Live

    The same scorer, pointed at real traffic

How a scorer runs before release and in production. Before release: dataset to task and In production: real traffic both reach one scorer, which emits pass · label · number and fans back out to experiment and live. experiment 91% pass rate, live 88% pass rate.

What you get

Every part of the eval workflow

  • A dataset list with a search field and a New dataset button. Hosted datasets read customer-support-qa, production-failures and golden-responses, each with its latest run and an experiment count; a code-defined agent-regression-suite follows.

    Datasets

    Build them in code or in the UI, or save a failing span straight from a trace. Every dataset keeps the experiment history.

  • A Recent events table filtered by evaluator. Rows list Flaky, Tone, LengthScore, ToolCorrectness and MaxDuration against results that are a pass, a label such as apologetic, a number such as 0.63, or an error, each with a timestamp.

    Evaluators

    Built-ins like Equals and MaxDuration, agent scorers like ToolCorrectness, or write your own. Each returns a pass, a label or a number.

  • An LLMJudge helpfulness panel for case_0042. Four judges score the same answer: sonnet-4 at 0.72, gpt-4o at 0.70, gemini-2.5 at 0.75 and grok-4 at 0.31 with a warning. A note reads that three of four judges agree the answer is helpful but omits the tracking link, and that grok-4 marked it unhelpful and flagged it for human review.

    LLM judges

    Score against a rubric, with the reasoning attached. When judges disagree, the run goes to a human.

  • A trace for invoke_agent billing_agent containing a chat gpt-4o span and an execute_tool check_stock span tagged used_stock_tool, with a SELECT stock FROM inventory query nested inside. Below it, a HasMatchingSpan evaluator queries for a span whose name contains check_stock.

    Assertions on spans

    Assert the tool actually fired, not just that the answer looked right. HasMatchingSpan queries the spans inside the run.

  • A Live evaluations table over the last 24 hours. billing_agent, support_agent and checkout_agent each carry Flaky, LengthScore and a third evaluator with their current values, an event count and how long ago the last event arrived.

    Live monitoring

    Score every agent as traffic arrives. Run the cheap checks on everything, the expensive judge on one percent.

  • An Annotate panel over a list of runs. Verdict offers Pass, Neutral and Fail on keys 1, 2 and 3, with Pass selected. A comment reads that the order is correct but the ship date is not in the trace, tags read accuracy and dates, and a footer notes two reviewers beside a Save and next button.

    Annotation queues

    Queue runs for a human verdict. Their answers become calibration data or new test cases.

Use cases

What people use Evaluate for

  • Turn a production failure into a test case

    Open the trace that went wrong, save it as a dataset case, and it becomes a regression test every future change has to pass.

    Evaluate an agent

  • Prove a change made things better

    Run the candidate against the baseline on the same dataset and read the difference per case, not as one blended average.

    Read the experiments guide

  • Show a regulator how the AI behaves

    Every run is a stored artifact with inputs, outputs, scores, and a timestamp. The eval becomes the release gate and the evidence.

    Review experiments

Customer stories

Teams who ship on evidence

FAQ

What is an eval?

A test for behavior with no single correct answer. Instead of asserting one exact output, you collect a dataset of cases, define scorers that judge each result, and track how the scores move when you change the prompt, model, or retrieval. In Logfire, eval results are stored as queryable telemetry next to the traces they scored, so you can query how a change scored with SQL.

What's the difference between offline and online evaluation?

Offline evaluation runs against a curated dataset while you develop, so it catches regressions before you deploy. Online evaluation scores real production traffic after it runs, so you catch regressions on inputs your dataset didn't contain yet. Most teams use both: offline to gate a release, online to watch what happens once it's live.

Do I need an eval suite to start?

No. Instrument your agent and Logfire records the runs you'll later turn into cases. Online evaluation scores real traffic without a curated dataset, so you get signal from day one. When you want to gate a release on numbers, save failing runs into a dataset and evaluate against it.

How is Logfire different from Braintrust for evals?

Both run evals, but Logfire keeps eval results as queryable telemetry next to the traces they scored, so a score links to the run that produced it. If you're already on Braintrust, the Braintrust SDK's Eval runs work against Logfire by changing two environment variables, for the standard Eval flow in Python and TypeScript. Braintrust-hosted datasets, prompts, and the model proxy are not covered. See Logfire vs Braintrust for the full comparison.

How do production failures become test cases?

A bad response in production is a trace in Logfire. Add it to a dataset with a click, and it becomes a case the next version has to pass. Over time the dataset records the actual ways your application has failed, so your test suite tracks real production behavior instead of cases you wrote up front.

Get started with Logfire for free

Sign up today and see your first trace in under 2 minutes

01 — Install 

$ uv add logfire

Resolved 7 packages in 128ms
Installed 7 packages in 89ms
+ logfire

02 — Instrument 

import logfire

logfire.configure()
logfire.info('Hello {name}', name='world')

Free up to 10m records per month No credit card needed