SDK Guide

The SDK provides a typed Python client for managing datasets programmatically. This is the recommended approach when you want to define schemas using Python types and automate dataset management. You can also manage datasets through the Web UI.

Installation

Install the Logfire SDK with the datasets extra:

Terminal

pip install 'logfire[datasets]'

This installs httpx and pydantic-evals as additional dependencies.

Creating a Client

from logfire.experimental.api_client import LogfireAPIClient

client = LogfireAPIClient(api_key='your-api-key')

The client can also be used as a context manager to ensure the underlying HTTP connection is properly closed:

with LogfireAPIClient(api_key='your-api-key') as client:
    ...

The base_url is automatically inferred from the API key. You can override it if needed (e.g., for self-hosters):

client = LogfireAPIClient(
    api_key='your-api-key',
    base_url='http://localhost:8000',
)

An async client is also available:

from logfire.experimental.api_client import AsyncLogfireAPIClient

async with AsyncLogfireAPIClient(api_key='your-api-key') as client:
    datasets = await client.list_datasets()

Creating a Dataset with Typed Schemas

Define your input, output, and metadata types as dataclasses or Pydantic models, then pass them to create_dataset. The SDK automatically generates JSON schemas from the types:

from dataclasses import dataclass

from logfire.experimental.api_client import LogfireAPIClient


@dataclass
class QuestionInput:
    question: str
    context: str | None = None


@dataclass
class AnswerOutput:
    answer: str
    confidence: float


@dataclass
class CaseMetadata:
    category: str
    difficulty: str
    reviewed: bool = False


with LogfireAPIClient(api_key='your-api-key') as client:
    dataset = client.create_dataset(
        name='qa-golden-set',
        description='Golden test cases for the Q&A system',
        input_type=QuestionInput,
        output_type=AnswerOutput,
        metadata_type=CaseMetadata,
        guidance='Each case should represent a realistic user question with a verified answer.',
    )
    print(f"Created dataset: {dataset['name']} (ID: {dataset['id']})")

All three type parameters are optional. You can create a dataset with just input_type, or with no types at all (in which case inputs and outputs are unvalidated JSON).

The guidance parameter lets you provide free-text instructions describing how cases should be structured.

Adding Cases

The SDK integrates directly with pydantic-evals Case objects. Use add_cases to add one or more cases:

from pydantic_evals import Case

client.add_cases(
    'qa-golden-set',
    cases=[
        Case(
            name='capital-question',
            inputs=QuestionInput(question='What is the capital of France?'),
            expected_output=AnswerOutput(answer='Paris', confidence=0.99),
            metadata=CaseMetadata(category='geography', difficulty='easy'),
        ),
        Case(
            name='math-question',
            inputs=QuestionInput(question='What is 15 * 23?'),
            expected_output=AnswerOutput(answer='345', confidence=1.0),
            metadata=CaseMetadata(category='math', difficulty='easy'),
        ),
    ],
    tags=['batch-import'],
)

You can also pass plain dicts instead of Case objects:

client.add_cases(
    'qa-golden-set',
    cases=[
        {'inputs': {'question': 'What color is the sky?'}, 'expected_output': {'answer': 'Blue'}},
    ],
)

Listing Cases

# List all cases in a dataset
cases = client.list_cases('qa-golden-set')
for case in cases:
    print(f"  {case['name']}: {case['inputs']}")

# Filter cases by tags
cases = client.list_cases('qa-golden-set', tags=['geography'])
for case in cases:
    print(f"  {case['name']}: {case['tags']}")

# Get a specific case
case = client.get_case('qa-golden-set', case_id='some-case-uuid')

Listing and Retrieving Datasets

# List all datasets in the project
datasets = client.list_datasets()
for ds in datasets:
    print(f"{ds['name']}: {ds['case_count']} cases")

# Get a specific dataset by name or ID
dataset_info = client.get_dataset('qa-golden-set', include_cases=False)

Updating and Deleting

# Update a dataset's metadata
client.update_dataset('qa-golden-set', description='Updated description')

# Update a specific case (including tags)
client.update_case(
    'qa-golden-set',
    case_id='some-case-uuid',
    metadata=CaseMetadata(category='geography', difficulty='easy', reviewed=True),
    tags=['verified', 'geography'],
)

# Delete a case
client.delete_case('qa-golden-set', case_id='some-case-uuid')

# Delete an entire dataset and all its cases
client.delete_dataset('qa-golden-set')

What’s Next?

Running Evaluations --- Fetch your dataset and run evaluations with pydantic-evals.
SDK Reference --- Complete method signatures and exception reference.
Web UI Guide --- Manage datasets through the Logfire web interface.