Skip to content

Require Verification

Send a coding agent back to verify its work when it tries to finish after editing code without a passing check since its last edit. Use it on agents that edit code, where a claim of finished, verified work should rest on a check that ran after the final edit.

While Pydantic AI Harness is on 0.x releases, the API may change between minor releases; when it does, deprecation warnings and release-note migration guidance tell you (or your agent) exactly how to upgrade. See the version policy.

Usage

Terminal
pip install pydantic-ai-harness
from pydantic_ai import Agent
from pydantic_ai_harness import FileSystem, RequireVerification, Shell

agent = Agent('anthropic:claude-opus-5-5', capabilities=[FileSystem(), Shell(), RequireVerification()])

RequireVerification records what the agent’s tool calls do during a run:

  • An edit is a successful FileSystem write_file or edit_file call, recognized by its tool kind whatever it is named, or a shell command matching edit_commands, such as ruff format . (but not ruff format --check .), sed -i, git checkout, mv, rm, an sh -c, python -c, or node -e script, or a > redirect. Edits that write_file or edit_file make to files matching exempt_paths don’t count. Shell edit commands always count, since the file they touch isn’t known.
  • A check is a shell command matching verification_commands, such as pytest, uv run --no-sync pyright, .venv/bin/pytest, ruff format --check ., pnpm -r test, or cargo build. The Shell tool’s result tells whether it passed.

Each edit makes every earlier check result stale. When the run would end after an edit, it is sent back unless at least one check has run since then and the latest run of each passed: an instruction carrying instructions is added to the conversation and the model answers again, with its tools available. After max_attempts of these redirects in one run, the run ends anyway and reports it with a CompletionUnverifiedEvent.

FieldDefaultMeaning
max_attempts2How many times per run the agent is sent back to verify
instructionsDEFAULT_REQUIRE_VERIFICATION_INSTRUCTIONSWhat the agent is told each time it is sent back, including how to run a check so it counts
verification_commandsDEFAULT_VERIFICATION_COMMANDSRegular expressions for the commands that count as checks
edit_commandsDEFAULT_EDIT_COMMANDSRegular expressions for the commands that count as edits
exempt_paths('*.md', '*.mdx', '*.rst')Glob patterns for the files whose edits need no check
from pydantic_ai_harness import RequireVerification

RequireVerification(
    max_attempts=1,
    instructions='Run `just ci` and report its result before you finish.',
    verification_commands=[r'just\s+ci\b'],
)

Recognizing commands

  • A command is read from the argument a tool’s metadata names with code_arg_name, on tools whose metadata sets code_arg_language='shell', as Shell’s run_command and shell tools do.
  • A command’s exit status is that of its last ;- or newline-separated statement, so a command counts as a check when one of the &&-joined steps of that statement starts with a match for a verification_commands pattern, after any leading NAME=value assignments. cd pkg; pytest counts, and pytest; echo done does not. A probe such as pytest --version, --help, or --collect-only is not a check. A statement with |, ||, a background &, or a command substitution does not count, and neither does a command with exit, exec, or return, because the exit status can then hide a failed check. File-descriptor duplications such as 2>&1, and separators and pipes inside quotes, are ignored for this.
  • A command counts as an edit when an edit_commands pattern matches anywhere in it outside quoted strings. The default > pattern skips ->, >&, and redirections to /dev/null. An edit command is not also a check, so ruff check --fix makes earlier checks stale and leaves the run with none.
  • A check passed when run_command reports no [exit code: N], timeout, or file-size limit notice, or when the shell tool reports an exit_code of 0. Results are read in the formats of Shell’s tools and of ACP’s run_command, which reports [exited with code N] or [terminated by signal S]. Only the shell tool’s result, recognized by its PID: ... (supervisor; ...) handles, is read for a JSON status, so a check whose own output ends in JSON still counts. A custom shell tool must end a failed command’s result with one of these lines, or its failures count as passes. A command started with start_command or the shell tool’s background mode, or still running when the shell tool returns, never counts as a check, even after it finishes, and clears the result of an earlier run of the same command. Run checks in the foreground.
  • Results are kept per command text, not per working directory. With Shell(persist_cwd=True), a later run of the same command in another directory replaces the earlier result. Use a command that names its directory, such as cd pkg && pytest, when the directory matters.
  • Results are read before Tool Output Limits truncates, spills, or summarizes them, wherever it is listed.
  • Calls from one model response run in parallel. A check that started before an edit in another call finished records no result, as it may have checked the code from before the edit.
  • Commands outside edit_commands are not edits. Extend it with the commands your agent uses to change files, such as a code generator.

Ending unverified

  • The first time the run would end unverified after every redirect is used, it emits a CompletionUnverifiedEvent with the number of attempts and the failed_commands, the checks whose latest run since the last edit failed. Listen for it with Hooks to fail a CI job or flag the result:
from pydantic_ai import Agent, RunContext
from pydantic_ai.capabilities import Hooks
from pydantic_ai_harness import FileSystem, RequireVerification, Shell
from pydantic_ai_harness.require_verification import CompletionUnverifiedEvent

unverified: list[CompletionUnverifiedEvent] = []
hooks = Hooks[None]()


@hooks.on.event(CompletionUnverifiedEvent)
async def record(ctx: RunContext[None], event: CompletionUnverifiedEvent) -> None:
    unverified.append(event)


agent = Agent('anthropic:claude-opus-5-5', capabilities=[FileSystem(), Shell(), RequireVerification(), hooks])
  • The event is dispatched immediately, so listeners get it before the run returns. It is emitted at most once per run, even when another capability, such as RefineOutput, continues the run after it. It is not recorded in the message history.
  • With max_attempts=0 the agent is never sent back, and an unverified run only emits the event.

Redirects

  • A redirect costs at least one more model request, more when the model runs checks during it, and counts toward UsageLimits like any other request.
  • The instruction is a VerificationRequiredPart, an InstructionCapabilityPart kind (a capability part in the application’s voice) whose attempt field holds the redirect number and never reaches the model. It keeps the default conversation scope, so it stays in the message history. Find redirects by isinstance(part, VerificationRequiredPart) (from pydantic_ai_harness.require_verification), not by their text.
  • Edits, check results, and attempts live in the instance for_run creates for each run, so each run starts with none and compaction does not reset them.
  • A run resumed after a pause for deferred tools, such as an approval, still knows about the successful write_file and edit_file edits made earlier in the same turn (not denied or failed ones): they are read from the message history. So are shell edit commands, for tools offered in the resumed run. Check results from before the pause are not, so the resumed run is sent back if it edited code earlier in the turn and has not run a check since resuming.
  • A model that answers a redirect without calling a tool, such as by explaining a blocker, is not sent back again: the run ends and emits the CompletionUnverifiedEvent.

Three kinds of run end without a redirect or an event:

  • A run that pauses for deferred tools and returns DeferredToolRequests, or ends while the provider is still processing a model request and returns PendingModelResponse.
  • A run whose output did not come from a model request, recognized by a response with no model_name.
  • Agent.run_stream(), which ends at the first output it streams. Use Agent.run() with an event_stream_handler, or Agent.iter().

Durable execution

Under durable execution, the tool hooks that record edits and checks and the hook that redirects the run (after_node_run) run in workflow code. When Temporal, DBOS, or Prefect replays a run, they run again in the same order over the journaled tool results and model responses, and send the run back the same way.

Telemetry

Each redirect records a zero-duration verification required span with require_verification.attempt. A run that ends unverified records a completion unverified span with require_verification.attempts. Neither carries command text.

Composition

  • Any edit makes every earlier check stale, including checks of code the edit cannot affect. Telling them apart would need each check’s full dependency closure (fixtures, configuration, imports), and a wrong guess would let unverified work through, so the capability prefers an extra check.
  • Edits made through other tools, such as a custom file tool, are not recorded.
  • Refine Output asks the agent to review its answer. RequireVerification asks it to run checks, and only after it edited code. With both on one agent, each sends the run back for its own reason.
  • Permissions may ask a person before a check runs. A denied check is not a check result.

See the source.

API reference

RequireVerification

Bases: AbstractCapability[AgentDepsT]

Send the agent back to verify its code edits before the run ends.

A successful FileSystem write_file or edit_file call, or a shell command matching edit_commands, is an edit and makes earlier verification results stale. A shell command matching verification_commands records whether it passed. When the run would end after an edit and the latest run of every verification command since then did not pass, a VerificationRequiredPart carrying instructions is added and the model answers again, up to max_attempts times per run. A run that still ends unverified emits a CompletionUnverifiedEvent.

from pydantic_ai import Agent
from pydantic_ai_harness import FileSystem, RequireVerification, Shell

agent = Agent('anthropic:claude-opus-5-5', capabilities=[FileSystem(), Shell(), RequireVerification()])

Attributes

max_attempts

How many times per run the agent is sent back to verify.

Type: int Default: 2

instructions

What the agent is told each time it is sent back.

Type: str Default: DEFAULT_REQUIRE_VERIFICATION_INSTRUCTIONS

verification_commands

Regular expressions for the commands that count as verification.

Type: Sequence[str] Default: DEFAULT_VERIFICATION_COMMANDS

edit_commands

Regular expressions for the commands that count as edits.

Type: Sequence[str] Default: DEFAULT_EDIT_COMMANDS

exempt_paths

Glob patterns for the files whose edits need no verification.

Type: Sequence[str] Default: ('*.md', '*.mdx', '*.rst')

Methods

get_ordering
def get_ordering() -> CapabilityOrdering

Run inside ToolOutputLimits, so results are read before it truncates, spills, or summarizes them.

Returns

CapabilityOrdering

prepare_tools

@async

def prepare_tools(
    ctx: RunContext[AgentDepsT],
    tool_defs: list[ToolDefinition],
) -> list[ToolDefinition]

Note which tools take a shell command, to read edits from the history of a resumed run.

Returns

list[ToolDefinition]

for_run

@async

def for_run(ctx: RunContext[AgentDepsT]) -> RequireVerification[AgentDepsT]

Return a fresh instance, so runs don’t share their edits, results, or attempts.

Returns

RequireVerification[AgentDepsT]

before_tool_execute

@async

def before_tool_execute(
    ctx: RunContext[AgentDepsT],
    *,
    call: ToolCallPart,
    tool_def: ToolDefinition,
    args: ValidatedToolArgs,
) -> ValidatedToolArgs

Note how many edits preceded a verification command, as calls in one response run in parallel.

Returns

ValidatedToolArgs

after_tool_execute

@async

def after_tool_execute(
    ctx: RunContext[AgentDepsT],
    *,
    call: ToolCallPart,
    tool_def: ToolDefinition,
    args: ValidatedToolArgs,
    result: Any,
) -> Any

Record a successful edit or verification command.

Returns

Any

after_node_run

@async

def after_node_run(
    ctx: RunContext[AgentDepsT],
    *,
    node: AgentNode[AgentDepsT],
    result: NodeResult[AgentDepsT],
) -> NodeResult[AgentDepsT]

Send an unverified run back to the model, or report that it ended unverified.

Returns

NodeResult[AgentDepsT]

CompletionUnverifiedEvent

Bases: CapabilityEvent

The run ended after code edits without passing verification, with every redirect used.

Dispatched immediately: it is emitted as the run ends, after its event stream has closed.

Attributes

attempts

How many times the run was sent back to verify before it ended.

Type: int

failed_commands

The verification commands whose latest run since the last edit failed.

Type: list[str]

VerificationRequiredPart

Bases: InstructionCapabilityPart

The instruction RequireVerification adds to the conversation when it sends the run back to verify.

Its content is the capability’s instructions, in the application’s voice. Find redirects in history by this type.

Attributes

attempt

Which of the run’s redirects it is, from 1. Never sent to the model.

Type: int

DEFAULT_REQUIRE_VERIFICATION_INSTRUCTIONS

What RequireVerification tells the agent each time it sends the run back, unless instructions is set.

Default: 'You edited code, but no verification has passed since your last edit. Run the relevant tests, lint, type checks, or build now, each as its own foreground command whose output is not piped or followed by another command, so its exit status shows whether it passed. If verification cannot run, explain the concrete blocker instead of claiming the work is verified.'

DEFAULT_VERIFICATION_COMMANDS

Regular expressions for the commands that count as verification, matched at the start of a command’s steps.

Type: tuple[str, …] Default: ('(?<!\\S)(?:(?:uv|poetry|pipenv|hatch|pdm)\\s+run(?:\\s+-\\S+)*\\s+)?(?:(?<!\\S)(?:\\S*/)?python3?\\s+-m\\s+)?(?<!\\S)(?:\\S*/)?(?:pytest|py\\.test|unittest|tox|nox|mypy|pyright|basedpyright|ty\\s+check|ruff\\s+check|(?:ruff\\s+format|black|isort)(?=[^;&|\\n]*\\s--(?:check|diff)))\\b', '(?:npm|pnpm|yarn|bun)(?:\\s+(?>workspace\\s+\\S+|-\\S+(?:\\s+(?!(?:run|test|lint|typecheck|check|build|workspace)\\b)[^-\\s]\\S*)?))*\\s+(?:run\\s+)?(?:test|lint|typecheck|check|build)\\b', '(?<!\\S)(?:npx\\s+)?(?<!\\S)(?:\\S*/)?(?:tsc|eslint|vitest|jest)\\b', '(?:cargo(?:\\s+\\+\\S+)?|go|dotnet)\\s+(?:test|build|check|clippy|vet)\\b', '(?:make|just)\\s+(?:\\S+\\s+)*?(?:test|lint|typecheck|check|build)\\w*\\b', '(?:mvn|gradle|\\./mvnw|\\./gradlew)\\s+(?:\\S+\\s+)*?(?:test|verify|check|build)\\b')

DEFAULT_EDIT_COMMANDS

Regular expressions for the commands that count as edits, searched for anywhere outside quoted strings.

Type: tuple[str, …] Default: ('(?:^|(?<=[;&|(\\n]))(?<![ \\t])[ \\t]*+(?:(?:uv|poetry|pipenv|hatch|pdm)\\s+run\\s+)?(?:ruff\\s+format|black|isort|cargo\\s+fmt|gofmt\\s+-w)\\b(?![^;&|\\n]*\\s--(?:check|diff))', '--(?:fix|write)\\b', '(?<![\\w-])(?:sed|perl)(?:\\s+-\\w+)*\\s+-\\w*i', '\\bgit\\s+(?:apply|am|checkout|cherry-pick|merge|mv|pull|rebase|reset|restore|revert|rm|stash|switch)\\b', '(?<![\\w-])(?:patch|mv|cp|rm|tee)\\s', '(?<![\\w-])(?:ba|z|da)?sh\\s+-\\w*c\\b', '(?<![\\w-])(?:python3?\\s+(?:-c|-)(?=\\s|$)|(?:node|ruby|perl)\\s+-e\\b)', '(?<![\\d&>-])>>?(?!&|\\s*/dev/null)')