pydantic_evals.evaluators
Bases: Generic[InputsT, OutputT, MetadataT]
Context for report-level evaluation, containing the full experiment results.
The experiment name.
Type: str
The full evaluation report.
Type: EvaluationReport[InputsT, OutputT, MetadataT]
Experiment-level metadata.
Bases: Generic[InputsT, OutputT, MetadataT]
Context for evaluating a task execution.
An instance of this class is the sole input to all Evaluators. It contains all the information needed to evaluate the task execution, including inputs, outputs, metadata, and telemetry data.
Evaluators use this context to access the task inputs, actual output, expected output, and other information when evaluating the result of the task execution.
Example:
from dataclasses import dataclass
from pydantic_evals.evaluators import Evaluator, EvaluatorContext
@dataclass
class ExactMatch(Evaluator):
def evaluate(self, ctx: EvaluatorContext) -> bool:
# Use the context to access task inputs, outputs, and expected outputs
return ctx.output == ctx.expected_output
The name of the case.
The inputs provided to the task for this case.
Type: InputsT
Metadata associated with the case, if provided. May be None if no metadata was specified.
Type: MetadataT | None
The expected output for the case, if provided. May be None if no expected output was specified.
Type: OutputT | None
The actual output produced by the task for this case.
Type: OutputT
The duration of the task run for this case.
Type: float
Attributes associated with the task run for this case.
These can be set by calling pydantic_evals.dataset.set_eval_attribute in any code executed
during the evaluation task.
Metrics associated with the task run for this case.
These can be set by calling pydantic_evals.dataset.increment_eval_metric in any code executed
during the evaluation task.
Get the SpanTree for this task execution.
The span tree is a graph where each node corresponds to an OpenTelemetry span recorded during the task execution, including timing information and any custom spans created during execution.
Type: SpanTree
Bases: Evaluator[object, object, object]
Check if the output exactly equals the provided value.
The result of running an evaluator with an optional explanation.
Contains a scalar value and an optional “reason” explaining the value.
The scalar result of the evaluation (boolean, integer, float, or string).
An optional explanation of the evaluation result.
Bases: BaseEvaluator, Generic[InputsT, OutputT, MetadataT]
Base class for experiment-wide evaluators that analyze full reports.
Unlike case-level Evaluators which assess individual task outputs, ReportEvaluators see all case results together and produce experiment-wide analyses like confusion matrices, precision-recall curves, or scalar statistics.
@abstractmethod
def evaluate(
ctx: ReportEvaluatorContext[InputsT, OutputT, MetadataT],
) -> ReportAnalysis | list[ReportAnalysis] | Awaitable[ReportAnalysis | list[ReportAnalysis]]
Evaluate the full report and return experiment-wide analysis/analyses.
ReportAnalysis | list[ReportAnalysis] | Awaitable[ReportAnalysis | list[ReportAnalysis]]
@async
def evaluate_async(
ctx: ReportEvaluatorContext[InputsT, OutputT, MetadataT],
) -> ReportAnalysis | list[ReportAnalysis]
Evaluate, handling both sync and async implementations.
ReportAnalysis | list[ReportAnalysis]
Bases: Evaluator[object, object, object]
Check if the output exactly equals the expected output.
Bases: Generic[EvaluationScalarT]
The details of an individual evaluation result.
Contains the name, value, reason, and source evaluator for a single evaluation.
name : str
The name of the evaluation.
The scalar result of the evaluation.
An optional explanation of the evaluation result.
The spec of the evaluator that produced this result.
Optional version tag for the evaluator that produced this result
(e.g. 'v2'). Sourced automatically from the evaluator’s
get_evaluator_version
method. Lets online-evaluation dashboards filter out results from retired versions
without deleting historical rows.
def downcast(*value_types: type[T]) -> EvaluationResult[T] | None
Attempt to downcast this result to a more specific type.
EvaluationResult[T] | None — A downcast version of this result if the value is an instance of one of the given types,
EvaluationResult[T] | None — otherwise None.
*value_types : type[T] Default: ()
The types to check the value against.
Bases: Evaluator[object, object, object]
Check if the output contains the expected output.
For strings, checks if expected_output is a substring of output. For lists/tuples, checks if expected_output is in output. For dicts, checks if all key-value pairs in expected_output are in output. For model-like types (BaseModel, dataclasses), converts to a dict and checks key-value pairs.
Note: case_sensitive only applies when both the value and output are strings.
Represents a failure raised during the execution of an evaluator.
Optional version tag for the evaluator that raised (e.g. 'v2'). Sourced automatically
from the evaluator’s
get_evaluator_version method.
Type: str | None Default: None
Class name of the exception that caused the failure (e.g. 'ValueError'). Populated
automatically when EvaluatorFailure is constructed from a caught exception; surfaced
as the error.type attribute on emitted OTel events.
Type: str | None Default: None
Bases: ReportEvaluator
Computes a confusion matrix from case data.
Bases: BaseEvaluator, Generic[InputsT, OutputT, MetadataT]
Base class for all evaluators.
Evaluators can assess the performance of a task in a variety of ways, as a function of the EvaluatorContext.
Subclasses must implement the evaluate method. Note it can be defined with either def or async def.
Example:
from dataclasses import dataclass
from pydantic_evals.evaluators import Evaluator, EvaluatorContext
@dataclass
class ExactMatch(Evaluator):
def evaluate(self, ctx: EvaluatorContext) -> bool:
return ctx.output == ctx.expected_output
Override get_default_evaluation_name
to customize the name used in reports, and
get_evaluator_version to tag the
evaluator with a version that downstream sinks can filter on.
Example:
from dataclasses import dataclass
from pydantic_evals.evaluators import Evaluator, EvaluatorContext
@dataclass
class LLMJudge(Evaluator):
def evaluate(self, ctx: EvaluatorContext) -> bool: ...
def get_evaluator_version(self) -> str | None:
return 'v2' # bumped after prompt rewrite
def get_default_evaluation_name() -> str
Return the default name to use in reports for the output of this evaluator.
Defaults to the serialization name of the evaluator (which is usually the class name). Override this method to customize the name, e.g. using instance information.
Note that evaluators that return a mapping of results will always use the keys of that mapping as the names of the associated evaluation results.
def get_evaluator_version() -> str | None
Return the version tag for this evaluator, or None if it has no version.
Propagated to online-evaluation sinks so dashboards can filter out results produced by retired
versions without deleting historical rows. Applies to every result the evaluator emits; bump
whenever behavior changes in a way that invalidates prior scores. Override this method to set
a non-None version.
@abstractmethod
def evaluate(
ctx: EvaluatorContext[InputsT, OutputT, MetadataT],
) -> EvaluatorOutput | Awaitable[EvaluatorOutput]
Evaluate the task output in the given context.
This is the main evaluation method that subclasses must implement. It can be either synchronous or asynchronous, returning either an EvaluatorOutput directly or an Awaitable[EvaluatorOutput].
EvaluatorOutput | Awaitable[EvaluatorOutput] — The evaluation result, which can be a scalar value, an EvaluationReason, or a mapping
EvaluatorOutput | Awaitable[EvaluatorOutput] — of evaluation names to either of those. Can be returned either synchronously or as an
EvaluatorOutput | Awaitable[EvaluatorOutput] — awaitable for asynchronous evaluation.
The context containing the inputs, outputs, and metadata for evaluation.
def evaluate_sync(ctx: EvaluatorContext[InputsT, OutputT, MetadataT]) -> EvaluatorOutput
Run the evaluator synchronously, handling both sync and async implementations.
This method ensures synchronous execution by running any async evaluate implementation to completion using run_until_complete.
EvaluatorOutput — The evaluation result, which can be a scalar value, an EvaluationReason, or a mapping
EvaluatorOutput — of evaluation names to either of those.
The context containing the inputs, outputs, and metadata for evaluation.
@async
def evaluate_async(
ctx: EvaluatorContext[InputsT, OutputT, MetadataT],
) -> EvaluatorOutput
Run the evaluator asynchronously, handling both sync and async implementations.
This method ensures asynchronous execution by properly awaiting any async evaluate implementation. For synchronous implementations, it returns the result directly.
EvaluatorOutput — The evaluation result, which can be a scalar value, an EvaluationReason, or a mapping
EvaluatorOutput — of evaluation names to either of those.
The context containing the inputs, outputs, and metadata for evaluation.
Bases: Evaluator[object, object, object]
Check if the output is an instance of a type with the given name.
Bases: ReportEvaluator
Computes a precision-recall curve from case data.
Returns both a PrecisionRecall chart and a ScalarResult with the AUC value.
The AUC is computed at full resolution (every unique score threshold) for accuracy,
while the chart points are downsampled to n_thresholds for display.
Bases: Evaluator[object, object, object]
Check if the execution time is under the specified maximum.
Bases: Evaluator[object, object, object]
Assert that the agent called a specific multiset of tools.
This compares the names of tools actually invoked (as a multiset) against
expected_tools. Repeated names require repeated calls — for example,
expected_tools=['search', 'search'] passes only if search was called
at least twice.
The tool names the agent is expected to call. Order does not matter; duplicates are significant.
allow_extra : bool Default: False
If False (the default), any tool call not listed in
expected_tools fails the check. Set to True to only require
that the expected tools were called, permitting extras.
include_failed : bool Default: False
If False (the default), tool-call attempts that
ended in an error (a raised exception, or a retry requested via
ModelRetry) are not counted. Set to True to count every
attempt.
Optional override for the reported evaluation name.
Returns EvaluationReason with a bool value.
Bases: TypedDict
Configuration for the score and assertion outputs of the LLMJudge evaluator.
Bases: Evaluator[object, object, object]
Judge whether the output of a language model meets the criteria of a provided rubric.
If you do not specify a model, it uses the default model for judging. This starts as ‘openai:gpt-5.2’, but can be
overridden by calling set_default_judge_model.
Bases: ReportEvaluator
Computes an ROC curve and AUC from case data.
Returns a LinePlot with the ROC curve (plus a dashed random-baseline diagonal)
and a ScalarResult with the AUC value.
Bases: Evaluator[object, object, object]
Compare the agent’s tool-call trajectory to an expected one.
The expected ordered list of tool names.
How strictly to compare:
'exact': actual must equal expected (1.0) or not (0.0).'in_order'(default): F1 computed from the longest common subsequence (LCS) of the two sequences. Extra calls reduce precision; missing calls reduce recall.'any_order': F1 computed from the multiset intersection of the two trajectories. Order is ignored, but extra and missing calls still reduce the score.
include_failed : bool Default: False
If False (the default), tool-call attempts that
ended in an error (a raised exception, or a retry requested via
ModelRetry) are not part of the trajectory. Set to True to
include every attempt.
Optional override for the reported evaluation name.
Returns EvaluationReason with a float value in [0.0, 1.0] (including
when no span tree was captured, in which case the value is 0.0). For the
F1-based modes, the reason text shows the precision, recall and F1
numbers so the score can be reproduced from the reported mismatch.
If both the expected and actual trajectories are empty, all modes score
1.0; if only one of them is empty, all modes score 0.0.
Bases: Evaluator[object, object, object]
G-Eval-style chain-of-thought evaluator (Liu et al., 2023).
The judge is shown the evaluation criteria and a list of explicit evaluation_steps,
produces a short reasoning trace, and emits an integer score within score_range (inclusive),
returned as an EvaluationReason. Because the
criteria and steps are user-supplied, GEval puts no structural requirements on ctx.inputs
or ctx.output.
If you do not specify a model, it uses the default model for judging. This starts as ‘openai:gpt-5.2’, but can be
overridden by calling set_default_judge_model.
Bases: ReportEvaluator
Computes a Kolmogorov-Smirnov plot and statistic from case data.
Plots the empirical CDFs of the score distribution for positive and negative cases, and computes the KS statistic (maximum vertical distance between the two CDFs).
Returns a LinePlot with the two CDF curves and a ScalarResult with the KS statistic.
Bases: Evaluator[object, object, object]
Check if the span tree contains a span that matches the specified query.
Bases: Evaluator[object, object, object]
Assert that a specific tool call received particular arguments.
Finds all local spans for tool_name in the run, picks the requested
occurrence, parses the recorded JSON arguments, and compares them to
expected_arguments.
tool_name : str
The tool whose arguments should be checked.
Expected argument keys/values.
'subset' (default) checks that every expected
key/value is present in the actual arguments. 'exact' requires
deep equality. Note that the subset comparison applies only to
top-level keys: an expected value (including a nested dict) must
compare equal to the actual value in full.
occurrence : ArgumentOccurrence | int Default: 'first'
Which invocation of the tool to inspect when the tool is
called multiple times: 'first', 'last', or a 0-based integer
index. A negative int is not supported.
include_failed : bool Default: False
If False (the default), tool-call attempts that
ended in an error (a raised exception, or a retry requested via
ModelRetry) are not considered. Set to True to consider every
attempt; each attempt then counts as a separate occurrence, so
'first' may select an attempt that was subsequently retried.
Optional override for the reported evaluation name.
Returns EvaluationReason with a bool value. Fails gracefully with a
descriptive reason if the tool was never called, the requested occurrence
doesn’t exist, or arguments weren’t recorded (e.g. include_content=False).
Bases: Evaluator[object, object, object]
Assert that the agent made at most max_calls locally-executed tool calls.
max_calls : int
Maximum allowed locally-executed tool calls.
include_failed : bool Default: True
If True (the default), tool-call attempts that ended
in an error (a raised exception, or a retry requested via
ModelRetry) count against the budget — they still consumed time
and tokens. Set to False to count only successful calls.
Optional override for the reported evaluation name.
Returns EvaluationReason with a bool value.
Bases: Evaluator[object, object, object]
Assert that the agent made at most max_requests model (chat) requests.
Prefers the requests value from ctx.metrics when available, otherwise
counts LLM request spans in the span tree directly (both use the same
criteria, so the two sources agree whenever both are populated).
max_requests : int
Maximum allowed model requests.
Optional override for the reported evaluation name.
Returns EvaluationReason with a bool value.
The specification of an evaluator to be run.
This class is used to represent evaluators in a serializable format, supporting various short forms for convenience when defining evaluators in YAML or JSON dataset files.
In particular, each of the following forms is supported for specifying an evaluator with name MyEvaluator:
'MyEvaluator'- Just the (string) name of the Evaluator subclass is used if its__init__takes no arguments{'MyEvaluator': first_arg}- A single argument is passed as the first positional argument toMyEvaluator.__init__{'MyEvaluator': {k1: v1, k2: v2}}- Multiple kwargs are passed toMyEvaluator.__init__
Default: NamedSpec
Type for the output of an evaluator, which can be a scalar, an EvaluationReason, or a mapping of names to either.
Default: EvaluationScalar | EvaluationReason | Mapping[str, EvaluationScalar | EvaluationReason]
How to compare the actual tool sequence to expected_trajectory.
'exact': actual must equal expected (1.0) or not (0.0).'in_order': F1 score combining precision and recall of the longest common subsequence.'any_order': F1 score combining precision and recall of the multiset intersection (order is ignored, but extra and missing calls both reduce the score).
Default: Literal['exact', 'in_order', 'any_order']
How to compare actual tool arguments to expected_arguments.
'exact': actual must deep-equal expected.'subset': every key/value in expected must be present (and equal) in actual.
Default: Literal['exact', 'subset']
Which occurrence of a tool call to inspect when a tool is called multiple times.
Default: Literal['first', 'last']
Bases: BaseModel
The output of a grading operation.
Bases: BaseModel
The output of a G-Eval grading operation.
G-Eval asks the judge to emit a short chain-of-thought reason followed by an
integer score in a user-specified range (see judge_g_eval).
@async
def judge_output(
output: Any,
rubric: str,
model: models.Model | models.KnownModelName | str | None = None,
model_settings: ModelSettings | None = None,
) -> GradingOutput
Judge the output of a model based on a rubric.
If the model is not specified, a default model is used. The default model starts as ‘openai:gpt-5.2’,
but this can be changed using the set_default_judge_model function.
GradingOutput
@async
def judge_input_output(
inputs: Any,
output: Any,
rubric: str,
model: models.Model | models.KnownModelName | str | None = None,
model_settings: ModelSettings | None = None,
) -> GradingOutput
Judge the output of a model based on the inputs and a rubric.
If the model is not specified, a default model is used. The default model starts as ‘openai:gpt-5.2’,
but this can be changed using the set_default_judge_model function.
GradingOutput
@async
def judge_input_output_expected(
inputs: Any,
output: Any,
expected_output: Any,
rubric: str,
model: models.Model | models.KnownModelName | str | None = None,
model_settings: ModelSettings | None = None,
) -> GradingOutput
Judge the output of a model based on the inputs and a rubric.
If the model is not specified, a default model is used. The default model starts as ‘openai:gpt-5.2’,
but this can be changed using the set_default_judge_model function.
GradingOutput
@async
def judge_output_expected(
output: Any,
expected_output: Any,
rubric: str,
model: models.Model | models.KnownModelName | str | None = None,
model_settings: ModelSettings | None = None,
) -> GradingOutput
Judge the output of a model based on the expected output, output, and a rubric.
If the model is not specified, a default model is used. The default model starts as ‘openai:gpt-5.2’,
but this can be changed using the set_default_judge_model function.
GradingOutput
def set_default_judge_model(model: models.Model | models.KnownModelName) -> None
Set the default model used for judging.
This model is used if None is passed to the model argument of judge_output and judge_input_output.
@async
def judge_g_eval(
output: Any,
criteria: str,
evaluation_steps: Sequence[str],
score_range: tuple[int, int] = (1, 5),
inputs: Any | None = None,
model: models.Model | models.KnownModelName | str | None = None,
model_settings: ModelSettings | None = None,
) -> GEvalOutput
Judge an output using a G-Eval style chain-of-thought prompt.
This is a simplified implementation of G-Eval (Liu et al., 2023, “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment”). The original paper computes an expectation over the distribution of score tokens using log-probs. We skip that step and simply ask the model for a direct integer score. This keeps the evaluator provider-agnostic at the cost of some correlation with human judgments.
GEvalOutput — A GEvalOutput containing
GEvalOutput — the judge’s reasoning and integer score.
output : Any
The output being evaluated.
criteria : str
The aspect being evaluated (e.g. “coherence”, “fluency”).
Explicit chain-of-thought steps the judge should follow.
Inclusive (min, max) integer score range.
Optional inputs/context to show alongside the output.
model : models.Model | models.KnownModelName | str | None Default: None
The model to use. If not specified, the default judge model is used.
model_settings : ModelSettings | None Default: None
Optional model settings.
ValueError— Ifscore_rangeis invalid,evaluation_stepsis empty, or the judge returns a score outside the range.