pydantic_evals.online_capability

Online evaluation capability for pydantic-ai agents.

Provides an OnlineEvaluation capability that attaches evaluators to agent runs, dispatching them asynchronously in the background after each run completes.

OnlineEvaluation

Bases: AbstractCapability[AgentDepsT]

Capability that runs online evaluators on agent run results.

Dispatches evaluators asynchronously in the background after each completed agent run. Non-blocking — the agent run returns without waiting for evaluators to finish.

Note

OnlineEvaluation wraps agent.run(), agent.run_stream(), and agent.iter() when the run reaches a final result. For streaming runs, evaluators are dispatched only after the final result is available and the surrounding context manager exits.

Under durable execution, sampling and dispatch run as one durable operation, so a replay doesn’t evaluate the run again. Pass an explicit id to use it with a durability capability.

Example:

from dataclasses import dataclass

from pydantic_ai import Agent
from pydantic_evals.evaluators import Evaluator, EvaluatorContext
from pydantic_evals.online_capability import OnlineEvaluation


@dataclass
class OutputNotEmpty(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> bool:
        return bool(ctx.output)


agent = Agent(
    'openai:gpt-5.2',
    name='assistant',
    capabilities=[OnlineEvaluation(evaluators=[OutputNotEmpty()])],
)

Attributes

evaluators

Evaluators to run after each agent run.

Plain functions are wrapped in a FunctionEvaluator.

Type: Sequence[Evaluator | EvaluatorFunc | OnlineEvaluator]

config

Optional config override. Defaults to the global DEFAULT_CONFIG.

Type: OnlineEvalConfig | None Default: None