Manage Datasets
Build a dataset, a stable collection of test cases that you can run repeatedly as your agent, model, or prompt changes.
Open a project, then select AI Evaluations > Datasets & experiments. The Datasets tab shows hosted and code-defined datasets in the selected time range.
Use the controls above the list to narrow a large project:
- Search for a dataset name.
- Filter Type to Hosted or Code-defined.
- Set Group by to Type, Path prefix, or None.
- Change the time range if an older dataset is missing.
Choose stable, descriptive names. Use names such as support/routing and support/response-quality when path-prefix grouping will help your team navigate related datasets.
Each row shows the actions available for that dataset. Edit cases is available when cases are hosted in Logfire. Review experiments opens the runs associated with that dataset.
If code creates many temporary dataset names, select Hide for me on entries you do not need. This changes only your view. Use Hidden above the list to restore them later.
Select New dataset, then choose where the source of truth should live:
- Manage in Logfire when teammates need to curate cases in the web UI or you want a shared hosted dataset.
- Manage in code when cases belong with the source and review process for your application.
If you manage the dataset in code, choose whether to Sync cases to Logfire or Keep cases in code. Experiment results can appear in Logfire with either choice when you configure Logfire for the run. Syncing also makes the cases available to browse and edit in Logfire.
The hosted-dataset flow has three steps:
- Dataset: enter a stable name and an optional description.
- Schemas: optionally define JSON schemas for inputs, expected outputs, and metadata.
- First case: optionally add an initial input, expected output, and metadata.
You can skip schemas and the first case. Add them later when the shape becomes clear.
After creation, use the Cases tab to maintain the dataset:
- Add case creates a case in the web UI.
- Add cases from code shows the code needed to publish cases.
- Sync cases from code imports code-defined cases into the hosted dataset.
- Export downloads the cases for reuse outside Logfire.
- Edit changes the dataset name, description, or schemas.
If Logfire already discovered a code-defined dataset with the same name, creating its hosted counterpart can import the latest cases instead of starting empty. Review the imported cases before relying on them as a shared test set.
Production failures and surprising outputs make useful regression cases:
- Open Live view and select the span you want to preserve.
- Select the database icon (+) in the span details.
- Choose an existing hosted dataset or create one.
- Review the extracted input, expected output, and metadata, then save the case.
The case keeps a link to its source trace, so reviewers can inspect the original behavior.
Before running an experiment, confirm that:
- the dataset has a stable name that future runs will reuse;
- hosted cases have the expected input and optional expected output;
- schemas describe the data you want teammates to enter;
- the dataset appears in the intended path-prefix group;
- Review experiments opens the expected run history.
This is expected when the cases remain only in code. Use Sync cases to Logfire, or create a hosted dataset with the same name and import the latest cases.
Widen the time range and clear the type filter. Also check Hidden if you previously hid the dataset for yourself.
Use Group by: Path prefix for consistently named datasets. Hide temporary entries for yourself, then change the code to reuse stable names for future runs.
Run an eval against the dataset, then review the experiment in Logfire.