Skip to content
Docs Pydantic Docs

Agentic Evaluators

Deterministic, span-based evaluators that grade an agent’s trajectory — the sequence and arguments of tool calls — rather than just its final output.

Agentic evaluators answer a class of “did the agent do the right thing?” questions that pure input/output checks can’t:

  • Tool coverage — did the agent call the specific tools it was supposed to? (ToolCorrectness)
  • Tool-call sequence — did it call them in the right order, or at least use the right set? (TrajectoryMatch)
  • Argument quality — did the tool receive the expected inputs? (ArgumentCorrectness)
  • Budget discipline — did the agent finish within a tool-call and/or model-request budget? (MaxToolCalls, MaxModelRequests)

They are all deterministic, never call an LLM, and are cheap enough to run on every case in every experiment.

ToolCorrectness

Assert that the agent called a specific multiset of tools. Repeated names require repeated calls.

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import ToolCorrectness

dataset = Dataset(
    name='rag_agent',
    cases=[Case(inputs='Summarize the latest papers on X')],
    evaluators=[
        ToolCorrectness(
            expected_tools=['search', 'rerank', 'generate'],
        ),
    ],
)

Parameters:

  • expected_tools (list[str]): Tool names the agent is expected to call. Order doesn’t matter; duplicates are significant — ['search', 'search'] requires two search calls.
  • allow_extra (bool, default False): By default, any tool call not listed in expected_tools fails the check. Set to True to only require that the expected tools were called, permitting extras.
  • include_failed (bool, default False): Whether to count tool-call attempts that ended in an error.
  • evaluation_name (str | None): Custom name in reports.

Returns: EvaluationReason with a bool value. The reason names missing and unexpected tools.

TrajectoryMatch

Compare the actual ordered list of tool names to an expected one, using one of three modes.

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import TrajectoryMatch

dataset = Dataset(
    name='ordered_tools',
    cases=[Case(inputs='Process and file this request')],
    evaluators=[
        TrajectoryMatch(
            expected_trajectory=['validate', 'enrich', 'submit'],
            order='in_order',
        ),
    ],
)

Parameters:

  • expected_trajectory (list[str]): Expected ordered list of tool names.
  • order (Literal['exact', 'in_order', 'any_order'], default 'in_order'):
    • 'exact' — 1.0 iff the sequences are equal, else 0.0.
    • 'in_order' — F1 computed from the longest common subsequence (LCS). Precision = LCS / len(actual), recall = LCS / len(expected). Allows extra calls interleaved with the expected order, but they reduce precision.
    • 'any_order' — F1 computed from the multiset intersection. Precision = overlap / len(actual), recall = overlap / len(expected). Order is ignored, but extra and missing calls both reduce the score.
  • include_failed (bool, default False): Whether the trajectory includes tool-call attempts that ended in an error.
  • evaluation_name (str | None): Custom name in reports.

Returns: EvaluationReason with a float value in [0.0, 1.0]. For the F1-based modes, the reason text spells out the overlap, precision, recall, and F1 so the score is reproducible from the mismatch.

For example, if expected = ['a', 'b', 'c'] and the agent called ['a', 'x', 'b'], the LCS is ['a', 'b'] (length 2), giving precision 2/3, recall 2/3, and F1 ≈ 0.667.

If both the expected and actual trajectories are empty, all modes score 1.0; if only one of them is empty, all modes score 0.0.

ArgumentCorrectness

Check that a specific tool call received particular arguments.

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import ArgumentCorrectness

dataset = Dataset(
    name='support_agent',
    cases=[Case(inputs='Refund order 12345')],
    evaluators=[
        ArgumentCorrectness(
            tool_name='issue_refund',
            expected_arguments={'order_id': '12345'},
            match_mode='subset',
            occurrence='first',
        ),
    ],
)

Parameters:

  • tool_name (str): The tool to inspect.
  • expected_arguments (dict[str, Any]): Expected argument keys/values.
  • match_mode (Literal['exact', 'subset'], default 'subset'):
    • 'subset' — every expected key/value is present in the actual arguments. Note that this applies only to top-level keys: an expected value (including a nested dict) must compare equal to the actual value in full.
    • 'exact' — deep equality; unexpected keys also fail.
  • occurrence (Literal['first', 'last'] | int, default 'first'): Which invocation to inspect if the tool is called multiple times. Integer indexes are 0-based.
  • include_failed (bool, default False): Whether tool-call attempts that ended in an error are considered. When True, each attempt counts as a separate occurrence.
  • evaluation_name (str | None): Custom name in reports.

Returns: EvaluationReason with a bool value.

Graceful degradation: this evaluator doesn’t crash when arguments aren’t available — for example, when the agent was instrumented with include_content=False, the evaluator returns False with a reason explaining the situation so your reports still make sense.

Checking arguments when telemetry excludes content

include_content=False removes tool arguments from the spans but keeps tool names, so ToolCorrectness, TrajectoryMatch, MaxToolCalls and MaxModelRequests work unchanged. Only ArgumentCorrectness fails, because it has no arguments to compare. If your deployment has to keep content out of telemetry, you have two options.

Read the arguments from the run’s messages. The messages an agent run returns hold every tool call with its arguments, whatever the instrumentation settings. Return them from your task alongside the output, and check them with a custom evaluator:

from dataclasses import dataclass
from typing import Any

from pydantic_ai import Agent, ModelMessage, ModelResponse, ToolCallPart
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EvaluationReason, Evaluator, EvaluatorContext

support_agent = Agent('openai:gpt-6.1-sol')


@support_agent.tool_plain
def issue_refund(order_id: str) -> str:
    return f'Refund issued for order {order_id}.'


@dataclass
class SupportRun:
    output: str
    messages: list[ModelMessage]


async def run_support_agent(query: str) -> SupportRun:
    result = await support_agent.run(query)
    return SupportRun(output=result.output, messages=result.all_messages())


@dataclass
class ToolCalledWith(Evaluator[str, SupportRun]):
    tool_name: str
    expected_arguments: dict[str, Any]

    def evaluate(self, ctx: EvaluatorContext[str, SupportRun]) -> EvaluationReason:
        calls = [
            part
            for message in ctx.output.messages
            if isinstance(message, ModelResponse)
            for part in message.parts
            if isinstance(part, ToolCallPart) and part.tool_name == self.tool_name
        ]
        if not calls:
            return EvaluationReason(value=False, reason=f'{self.tool_name!r} was not called.')
        arguments = [call.args_as_dict() for call in calls]
        matches = any(
            all(key in args and args[key] == value for key, value in self.expected_arguments.items())
            for args in arguments
        )
        return EvaluationReason(value=matches, reason=f'Arguments: {arguments!r}')


dataset = Dataset(
    name='refund_arguments',
    cases=[Case(inputs='Refund order 12345')],
    evaluators=[ToolCalledWith(tool_name='issue_refund', expected_arguments={'order_id': '12345'})],
)

The messages record each call the model made, so unlike ArgumentCorrectness, which skips failed or retried attempts by default and never sees deferred calls, they include all of these; this evaluator passes if any call has the expected arguments. Evaluators that check the output, such as LLMJudge, see the whole SupportRun.

Capture content only where you run evals. The span-based evaluators read the spans of the process that runs the dataset, so that process can instrument with content while production doesn’t. Agent.instrument_all() sets the default for agents without instrumentation settings of their own: call Agent.instrument_all(InstrumentationSettings(include_content=True)) in your eval script, with InstrumentationSettings imported from pydantic_ai, and include_content=False in production. An agent with settings of its own, from an Instrumentation capability or its instrument attribute, keeps them, so build those settings from configuration instead of hard-coding them. The spans with content are exported to wherever the eval process sends its telemetry.

MaxToolCalls and MaxModelRequests

Assert that the agent stayed within a tool-call and/or model-request budget. These work like MaxDuration: one budget per evaluator, each reported as its own boolean assertion.

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import MaxModelRequests, MaxToolCalls

dataset = Dataset(
    name='budget_aware',
    cases=[Case(inputs='Draft a short reply')],
    evaluators=[
        MaxToolCalls(max_calls=5),
        MaxModelRequests(max_requests=3),
    ],
)

Parameters:

  • MaxToolCalls: max_calls (int) — maximum allowed locally-executed tool calls. include_failed (bool, default True) controls whether attempts that ended in an error count against the budget (by default they do — they still consumed time and tokens).
  • MaxModelRequests: max_requests (int) — maximum allowed model (chat) requests. Prefers the requests value from ctx.metrics when available, otherwise counts LLM request spans directly (both use the same criteria).
  • Both accept evaluation_name (str | None) to customize the name in reports — useful when the same budget check appears at both the dataset and case level.

Returns: EvaluationReason with a bool value. The reason includes the observed count and the budget.

Recipes

RAG agent

Check that the retrieval pipeline runs search → rerank → generate, with no unexpected tool calls.

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import ToolCorrectness, TrajectoryMatch

dataset = Dataset(
    name='rag_pipeline',
    cases=[Case(inputs='Find papers on in-context learning')],
    evaluators=[
        ToolCorrectness(
            expected_tools=['search', 'rerank', 'generate'],
        ),
        TrajectoryMatch(
            expected_trajectory=['search', 'rerank', 'generate'],
            order='exact',
        ),
    ],
)

Multi-tool agent where order matters

Allow occasional retries, but require the main steps to happen in order.

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import TrajectoryMatch

dataset = Dataset(
    name='ordered_with_slack',
    cases=[Case(inputs='Process shipment 99')],
    evaluators=[
        TrajectoryMatch(
            expected_trajectory=['validate', 'enrich', 'submit'],
            order='in_order',  # F1-based: extra calls only reduce precision, order must be preserved
        ),
    ],
)

Support agent with ArgumentCorrectness and budget checks

Verify that the right action was taken with the right inputs — within a reasonable number of steps.

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import (
    ArgumentCorrectness,
    MaxModelRequests,
    MaxToolCalls,
)

dataset = Dataset(
    name='refund_handling',
    cases=[
        Case(
            name='valid_refund',
            inputs={'query': 'Refund my order', 'order_id': '12345'},
            evaluators=[
                ArgumentCorrectness(
                    tool_name='issue_refund',
                    expected_arguments={'order_id': '12345'},
                ),
            ],
        ),
    ],
    evaluators=[
        MaxToolCalls(max_calls=4),
        MaxModelRequests(max_requests=2),
    ],
)

Task completion judged with the tool-call trajectory

For tasks where deterministic checks aren’t enough, you can have an LLM judge the task outcome together with the tool-call trajectory. LLMJudge only sees the case inputs, output, and expected output — not other evaluators’ results or the span tree — so to give the judge visibility into how the agent got there, write a small custom evaluator that extracts the trajectory from the span tree and passes it to judge_input_output directly:

from dataclasses import dataclass

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EvaluationReason, Evaluator, EvaluatorContext
from pydantic_evals.evaluators.llm_as_a_judge import judge_input_output
from pydantic_evals.otel import SpanTreeRecordingError


@dataclass
class TrajectoryJudge(Evaluator):
    rubric: str

    async def evaluate(self, ctx: EvaluatorContext) -> EvaluationReason:
        try:
            span_tree = ctx.span_tree
        except SpanTreeRecordingError:
            # Degrade gracefully, like the built-in evaluators on this page.
            return EvaluationReason(value=False, reason='No span tree available.')

        # Build a plain-text trajectory summary, mirroring what the built-in
        # evaluators count as a tool call by default: tool spans are named
        # 'running tool' (v2) or 'execute_tool {name}' (v3+); deferred calls
        # never ran; output functions share the tool span shape but aren't
        # tool calls; and failed attempts (status 'error') are dropped, like
        # the built-in evaluators' `include_failed=False` default.
        tool_names = [
            node.attributes['gen_ai.tool.name']
            for node in span_tree
            if 'gen_ai.tool.name' in node.attributes
            and 'pydantic_ai.tool.deferral.name' not in node.attributes
            and node.status != 'error'
            and (node.name == 'running tool' or node.name.startswith('execute_tool '))
            and not str(node.attributes.get('logfire.msg', '')).startswith('running output function:')
        ]
        trajectory = ', '.join(str(n) for n in tool_names) or '(none)'
        grading_output = await judge_input_output(
            {'query': ctx.inputs, 'tool_trajectory': trajectory},
            ctx.output,
            self.rubric,
        )
        return EvaluationReason(value=grading_output.pass_, reason=grading_output.reason)


dataset = Dataset(
    name='task_completion',
    cases=[Case(inputs='Resolve ticket 42')],
    evaluators=[
        TrajectoryJudge(
            rubric=(
                'The agent completed the task correctly, and the tool trajectory '
                'included in the input is reasonable for the given query.'
            ),
        ),
    ],
)

This pattern keeps the deterministic checks above cheap and reproducible, and reserves the qualitative, open-ended judgement for the LLM — with the trajectory explicitly included in what the judge sees.

Next steps