Require Verification
Send a coding agent back to verify its work when it tries to finish after editing code without a passing check since its last edit. Use it on agents that edit code, where a claim of finished, verified work should rest on a check that ran after the final edit.
While Pydantic AI Harness is on 0.x releases, the API may change between minor releases; when it does, deprecation warnings and release-note migration guidance tell you (or your agent) exactly how to upgrade. See the version policy.
pip install pydantic-ai-harness
uv add pydantic-ai-harness
from pydantic_ai import Agent
from pydantic_ai_harness import FileSystem, RequireVerification, Shell
agent = Agent('anthropic:claude-opus-5-5', capabilities=[FileSystem(), Shell(), RequireVerification()])
RequireVerification records what the agent’s tool calls do during a run:
- An edit is a successful
FileSystemwrite_fileoredit_filecall, recognized by its tool kind whatever it is named, or a shell command matchingedit_commands, such asruff format .(but notruff format --check .),sed -i,git checkout,mv,rm, ansh -c,python -c, ornode -escript, or a>redirect. Edits thatwrite_fileoredit_filemake to files matchingexempt_pathsdon’t count. Shell edit commands always count, since the file they touch isn’t known. - A check is a shell command matching
verification_commands, such aspytest,uv run --no-sync pyright,.venv/bin/pytest,ruff format --check .,pnpm -r test, orcargo build. TheShelltool’s result tells whether it passed.
Each edit makes every earlier check result stale. When the run would end after an edit, it is sent back unless at least one check has run since then and the latest run of each passed: an instruction carrying instructions is added to the conversation and the model answers again, with its tools available. After max_attempts of these redirects in one run, the run ends anyway and reports it with a CompletionUnverifiedEvent.
| Field | Default | Meaning |
|---|---|---|
max_attempts | 2 | How many times per run the agent is sent back to verify |
instructions | DEFAULT_REQUIRE_VERIFICATION_INSTRUCTIONS | What the agent is told each time it is sent back, including how to run a check so it counts |
verification_commands | DEFAULT_VERIFICATION_COMMANDS | Regular expressions for the commands that count as checks |
edit_commands | DEFAULT_EDIT_COMMANDS | Regular expressions for the commands that count as edits |
exempt_paths | ('*.md', '*.mdx', '*.rst') | Glob patterns for the files whose edits need no check |
from pydantic_ai_harness import RequireVerification
RequireVerification(
max_attempts=1,
instructions='Run `just ci` and report its result before you finish.',
verification_commands=[r'just\s+ci\b'],
)
- A command is read from the argument a tool’s metadata names with
code_arg_name, on tools whose metadata setscode_arg_language='shell', asShell’srun_commandandshelltools do. - A command’s exit status is that of its last
;- or newline-separated statement, so a command counts as a check when one of the&&-joined steps of that statement starts with a match for averification_commandspattern, after any leadingNAME=valueassignments.cd pkg; pytestcounts, andpytest; echo donedoes not. A probe such aspytest --version,--help, or--collect-onlyis not a check. A statement with|,||, a background&, or a command substitution does not count, and neither does a command withexit,exec, orreturn, because the exit status can then hide a failed check. File-descriptor duplications such as2>&1, and separators and pipes inside quotes, are ignored for this. - A command counts as an edit when an
edit_commandspattern matches anywhere in it outside quoted strings. The default>pattern skips->,>&, and redirections to/dev/null. An edit command is not also a check, soruff check --fixmakes earlier checks stale and leaves the run with none. - A check passed when
run_commandreports no[exit code: N], timeout, or file-size limit notice, or when theshelltool reports anexit_codeof0. Results are read in the formats ofShell’s tools and of ACP’srun_command, which reports[exited with code N]or[terminated by signal S]. Only theshelltool’s result, recognized by itsPID: ... (supervisor; ...)handles, is read for a JSON status, so a check whose own output ends in JSON still counts. A custom shell tool must end a failed command’s result with one of these lines, or its failures count as passes. A command started withstart_commandor theshelltool’s background mode, or still running when theshelltool returns, never counts as a check, even after it finishes, and clears the result of an earlier run of the same command. Run checks in the foreground. - Results are kept per command text, not per working directory. With
Shell(persist_cwd=True), a later run of the same command in another directory replaces the earlier result. Use a command that names its directory, such ascd pkg && pytest, when the directory matters. - Results are read before Tool Output Limits truncates, spills, or summarizes them, wherever it is listed.
- Calls from one model response run in parallel. A check that started before an edit in another call finished records no result, as it may have checked the code from before the edit.
- Commands outside
edit_commandsare not edits. Extend it with the commands your agent uses to change files, such as a code generator.
- The first time the run would end unverified after every redirect is used, it emits a
CompletionUnverifiedEventwith the number ofattemptsand thefailed_commands, the checks whose latest run since the last edit failed. Listen for it withHooksto fail a CI job or flag the result:
from pydantic_ai import Agent, RunContext
from pydantic_ai.capabilities import Hooks
from pydantic_ai_harness import FileSystem, RequireVerification, Shell
from pydantic_ai_harness.require_verification import CompletionUnverifiedEvent
unverified: list[CompletionUnverifiedEvent] = []
hooks = Hooks[None]()
@hooks.on.event(CompletionUnverifiedEvent)
async def record(ctx: RunContext[None], event: CompletionUnverifiedEvent) -> None:
unverified.append(event)
agent = Agent('anthropic:claude-opus-5-5', capabilities=[FileSystem(), Shell(), RequireVerification(), hooks])
- The event is dispatched immediately, so listeners get it before the run returns. It is emitted at most once per run, even when another capability, such as
RefineOutput, continues the run after it. It is not recorded in the message history. - With
max_attempts=0the agent is never sent back, and an unverified run only emits the event.
- A redirect costs at least one more model request, more when the model runs checks during it, and counts toward
UsageLimitslike any other request. - The instruction is a
VerificationRequiredPart, anInstructionCapabilityPartkind (a capability part in the application’s voice) whoseattemptfield holds the redirect number and never reaches the model. It keeps the default conversation scope, so it stays in the message history. Find redirects byisinstance(part, VerificationRequiredPart)(frompydantic_ai_harness.require_verification), not by their text. - Edits, check results, and attempts live in the instance
for_runcreates for each run, so each run starts with none and compaction does not reset them. - A run resumed after a pause for deferred tools, such as an approval, still knows about the successful
write_fileandedit_fileedits made earlier in the same turn (not denied or failed ones): they are read from the message history. So are shell edit commands, for tools offered in the resumed run. Check results from before the pause are not, so the resumed run is sent back if it edited code earlier in the turn and has not run a check since resuming. - A model that answers a redirect without calling a tool, such as by explaining a blocker, is not sent back again: the run ends and emits the
CompletionUnverifiedEvent.
Three kinds of run end without a redirect or an event:
- A run that pauses for deferred tools and returns
DeferredToolRequests, or ends while the provider is still processing a model request and returnsPendingModelResponse. - A run whose output did not come from a model request, recognized by a response with no
model_name. Agent.run_stream(), which ends at the first output it streams. UseAgent.run()with anevent_stream_handler, orAgent.iter().
Under durable execution, the tool hooks that record edits and checks and the hook that redirects the run (after_node_run) run in workflow code. When Temporal, DBOS, or Prefect replays a run, they run again in the same order over the journaled tool results and model responses, and send the run back the same way.
Each redirect records a zero-duration verification required span with require_verification.attempt. A run that ends unverified records a completion unverified span with require_verification.attempts. Neither carries command text.
- Any edit makes every earlier check stale, including checks of code the edit cannot affect. Telling them apart would need each check’s full dependency closure (fixtures, configuration, imports), and a wrong guess would let unverified work through, so the capability prefers an extra check.
- Edits made through other tools, such as a custom file tool, are not recorded.
- Refine Output asks the agent to review its answer.
RequireVerificationasks it to run checks, and only after it edited code. With both on one agent, each sends the run back for its own reason. - Permissions may ask a person before a check runs. A denied check is not a check result.
See the source.
Bases: AbstractCapability[AgentDepsT]
Send the agent back to verify its code edits before the run ends.
A successful FileSystem write_file or edit_file call, or a shell command matching
edit_commands, is an edit and makes earlier verification results stale. A shell command matching
verification_commands records whether it passed. When the run would end after an edit and the
latest run of every verification command since then did not pass, a
VerificationRequiredPart carrying
instructions is added and the model answers again, up to max_attempts times per run. A run that
still ends unverified emits a CompletionUnverifiedEvent.
from pydantic_ai import Agent
from pydantic_ai_harness import FileSystem, RequireVerification, Shell
agent = Agent('anthropic:claude-opus-5-5', capabilities=[FileSystem(), Shell(), RequireVerification()])
How many times per run the agent is sent back to verify.
Type: int Default: 2
What the agent is told each time it is sent back.
Type: str Default: DEFAULT_REQUIRE_VERIFICATION_INSTRUCTIONS
Regular expressions for the commands that count as verification.
Type: Sequence[str] Default: DEFAULT_VERIFICATION_COMMANDS
Regular expressions for the commands that count as edits.
Type: Sequence[str] Default: DEFAULT_EDIT_COMMANDS
Glob patterns for the files whose edits need no verification.
Type: Sequence[str] Default: ('*.md', '*.mdx', '*.rst')
def get_ordering() -> CapabilityOrdering
Run inside ToolOutputLimits, so results are read before it truncates, spills, or summarizes them.
CapabilityOrdering
@async
def prepare_tools(
ctx: RunContext[AgentDepsT],
tool_defs: list[ToolDefinition],
) -> list[ToolDefinition]
Note which tools take a shell command, to read edits from the history of a resumed run.
@async
def for_run(ctx: RunContext[AgentDepsT]) -> RequireVerification[AgentDepsT]
Return a fresh instance, so runs don’t share their edits, results, or attempts.
RequireVerification[AgentDepsT]
@async
def before_tool_execute(
ctx: RunContext[AgentDepsT],
*,
call: ToolCallPart,
tool_def: ToolDefinition,
args: ValidatedToolArgs,
) -> ValidatedToolArgs
Note how many edits preceded a verification command, as calls in one response run in parallel.
ValidatedToolArgs
@async
def after_tool_execute(
ctx: RunContext[AgentDepsT],
*,
call: ToolCallPart,
tool_def: ToolDefinition,
args: ValidatedToolArgs,
result: Any,
) -> Any
Record a successful edit or verification command.
@async
def after_node_run(
ctx: RunContext[AgentDepsT],
*,
node: AgentNode[AgentDepsT],
result: NodeResult[AgentDepsT],
) -> NodeResult[AgentDepsT]
Send an unverified run back to the model, or report that it ended unverified.
NodeResult[AgentDepsT]
Bases: CapabilityEvent
The run ended after code edits without passing verification, with every redirect used.
Dispatched immediately: it is emitted as the run ends, after its event stream has closed.
How many times the run was sent back to verify before it ended.
Type: int
The verification commands whose latest run since the last edit failed.
Bases: InstructionCapabilityPart
The instruction RequireVerification adds to the conversation when it sends the run back to verify.
Its content is the capability’s instructions, in the application’s voice. Find redirects in history by this
type.
Which of the run’s redirects it is, from 1. Never sent to the model.
Type: int
What RequireVerification tells the agent each time it sends the run back, unless instructions is set.
Default: 'You edited code, but no verification has passed since your last edit. Run the relevant tests, lint, type checks, or build now, each as its own foreground command whose output is not piped or followed by another command, so its exit status shows whether it passed. If verification cannot run, explain the concrete blocker instead of claiming the work is verified.'
Regular expressions for the commands that count as verification, matched at the start of a command’s steps.
Type: tuple[str, …] Default: ('(?<!\\S)(?:(?:uv|poetry|pipenv|hatch|pdm)\\s+run(?:\\s+-\\S+)*\\s+)?(?:(?<!\\S)(?:\\S*/)?python3?\\s+-m\\s+)?(?<!\\S)(?:\\S*/)?(?:pytest|py\\.test|unittest|tox|nox|mypy|pyright|basedpyright|ty\\s+check|ruff\\s+check|(?:ruff\\s+format|black|isort)(?=[^;&|\\n]*\\s--(?:check|diff)))\\b', '(?:npm|pnpm|yarn|bun)(?:\\s+(?>workspace\\s+\\S+|-\\S+(?:\\s+(?!(?:run|test|lint|typecheck|check|build|workspace)\\b)[^-\\s]\\S*)?))*\\s+(?:run\\s+)?(?:test|lint|typecheck|check|build)\\b', '(?<!\\S)(?:npx\\s+)?(?<!\\S)(?:\\S*/)?(?:tsc|eslint|vitest|jest)\\b', '(?:cargo(?:\\s+\\+\\S+)?|go|dotnet)\\s+(?:test|build|check|clippy|vet)\\b', '(?:make|just)\\s+(?:\\S+\\s+)*?(?:test|lint|typecheck|check|build)\\w*\\b', '(?:mvn|gradle|\\./mvnw|\\./gradlew)\\s+(?:\\S+\\s+)*?(?:test|verify|check|build)\\b')
Regular expressions for the commands that count as edits, searched for anywhere outside quoted strings.
Type: tuple[str, …] Default: ('(?:^|(?<=[;&|(\\n]))(?<![ \\t])[ \\t]*+(?:(?:uv|poetry|pipenv|hatch|pdm)\\s+run\\s+)?(?:ruff\\s+format|black|isort|cargo\\s+fmt|gofmt\\s+-w)\\b(?![^;&|\\n]*\\s--(?:check|diff))', '--(?:fix|write)\\b', '(?<![\\w-])(?:sed|perl)(?:\\s+-\\w+)*\\s+-\\w*i', '\\bgit\\s+(?:apply|am|checkout|cherry-pick|merge|mv|pull|rebase|reset|restore|revert|rm|stash|switch)\\b', '(?<![\\w-])(?:patch|mv|cp|rm|tee)\\s', '(?<![\\w-])(?:ba|z|da)?sh\\s+-\\w*c\\b', '(?<![\\w-])(?:python3?\\s+(?:-c|-)(?=\\s|$)|(?:node|ruby|perl)\\s+-e\\b)', '(?<![\\d&>-])>>?(?!&|\\s*/dev/null)')