pydantic_evals.online_capability

Online evaluation capability for pydantic-ai agents.

Provides an OnlineEvaluation capability that attaches evaluators to agent runs, dispatching them asynchronously in the background after each run completes.

OnlineEvaluation

Bases: AbstractCapability[AgentDepsT]

Capability that runs online evaluators on agent run results.

Dispatches evaluators asynchronously in the background after each completed agent run. Non-blocking — the agent run returns without waiting for evaluators to finish.

Note

OnlineEvaluation wraps agent.run(), agent.run_stream(), and agent.iter() when the run reaches a final result. For streaming runs, evaluators are dispatched only after the final result is available and the surrounding context manager exits.

Example:

from dataclasses import dataclass

from pydantic_ai import Agent
from pydantic_evals.evaluators import Evaluator, EvaluatorContext
from pydantic_evals.online_capability import OnlineEvaluation


@dataclass
class OutputNotEmpty(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> bool:
        return bool(ctx.output)


agent = Agent(
    'openai:gpt-5.2',
    name='assistant',
    capabilities=[OnlineEvaluation(evaluators=[OutputNotEmpty()])],
)

Attributes

evaluators

Evaluators to run after each agent run.

Type: Sequence[Evaluator | OnlineEvaluator]

config

Optional config override. Defaults to the global DEFAULT_CONFIG.

Type: OnlineEvalConfig | None Default: None