Skip to content

Retry On Model Errors

RetryOnModelErrors is a capability that attempts the same model again when a request to it fails with an error that waiting may fix, like a rate limit, an overloaded model or a dropped connection:

retry_on_model_errors.py
from pydantic_ai import Agent
from pydantic_ai.capabilities import RetryOnModelErrors

agent = Agent('openai:gpt-6.1-sol', capabilities=[RetryOnModelErrors()])

It’s the agent’s own retry layer: it sees the normalized model errors every model raises rather than status codes, retries a stream that fails before any output reaches you, and records every attempt on the response that answers. See Retries for how it relates to the other layers that retry.

Retry or fall back

RetryOnModelErrors attempts the same model again, for a failure that is likely to be gone a moment later. ModelFallback moves on to a different model, for a failure that won’t be, like a model that’s down, an exhausted quota, a request the model rejects, or a refusal. They’re separate capabilities that compose: with both, each model is retried before the request falls back to the next one, and the next model gets a fresh budget of attempts of its own.

retry_then_fall_back.py
from pydantic_ai import Agent, ModelMessage, ModelResponse, TextPart
from pydantic_ai.capabilities import ModelFallback, RetryOnModelErrors
from pydantic_ai.exceptions import ModelOverloadedError
from pydantic_ai.models.function import AgentInfo, FunctionModel


def overloaded(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
    raise ModelOverloadedError('primary', 'Overloaded', retry_after=0.001)


def answer(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
    return ModelResponse(parts=[TextPart('The capital of France is Paris.')])


agent = Agent(
    # These deterministic models stand in for two providers.
    FunctionModel(overloaded, model_name='primary'),
    capabilities=[
        ModelFallback(FunctionModel(answer, model_name='backup')),
        RetryOnModelErrors(max_attempts=3),
    ],
)
result = agent.run_sync('What is the capital of France?')
print(result.output)
#> The capital of France is Paris.
for attempt in result.response.failed_attempts or []:
    print(attempt.model_name, attempt.kind, attempt.retry_after)
    #> primary retry 0.001
    #> primary retry 0.001
    #> primary fallback 0.001

The order you list them in doesn’t matter: RetryOnModelErrors always judges an attempt’s failure before ModelFallback does. An error RetryOnModelErrors doesn’t retry, or stops retrying, goes on to ModelFallback’s fallback_on as it would without it.

When to retry

By default, a request is retried when it fails with one of these error categories, which TRANSIENT_MODEL_ERRORS lists:

Errors that fail the same way when the request is sent again aren’t retried by default: a ModelRequestRejectedError (a bad request, bad credentials, an unknown model or an overflowing context window), a ModelQuotaExceededError, a ModelRequestTimeout for a request that ran out its whole request_timeout, or a plain ModelAPIError that falls in no category.

Nor is a refusal, a response with a finish_reason of 'content_filter': the same model usually refuses the same request again, which is why ModelFallback moves on to another model when one refuses. To retry a refusal on the same model all the same, name ContentFilterError in retry_on. As in fallback_on, it then stands for any refused response, empty or with partial text, because the agent raises the error only after it has accepted a response, too late to retry it. Once the model’s attempts are used up, the last refusal is accepted, and a ModelFallback can still move on from it. Likewise, naming OutputTokenLimitExceeded retries a response the output token limit cut off (finish_reason='length').

The retry_on parameter chooses differently. It takes the same forms as ModelFallback’s fallback_on:

  • a tuple of exception types
  • an exception handler, (Exception) -> bool
  • a response handler, (ModelResponse) -> bool, which rejects a response that was returned successfully, so the same model is asked again
  • a sequence mixing any of the above
retry_on_rate_limits.py
from pydantic_ai import Agent
from pydantic_ai.capabilities import RetryOnModelErrors
from pydantic_ai.exceptions import ModelOverloadedError, ModelRateLimitError

agent = Agent(
    'anthropic:claude-sonnet-5-5',
    capabilities=[RetryOnModelErrors(retry_on=(ModelRateLimitError, ModelOverloadedError), max_attempts=5)],
)

To retry another error on top of the default ones, extend TRANSIENT_MODEL_ERRORS. The same tuple tells you whether an error is one of them, as in isinstance(error, TRANSIENT_MODEL_ERRORS):

retry_on_transient_and_quota.py
from pydantic_ai import Agent
from pydantic_ai.capabilities import TRANSIENT_MODEL_ERRORS, RetryOnModelErrors
from pydantic_ai.exceptions import ModelQuotaExceededError

agent = Agent(
    'openai:gpt-6.1-sol',
    capabilities=[RetryOnModelErrors(retry_on=(*TRANSIENT_MODEL_ERRORS, ModelQuotaExceededError))],
)

Whatever retry_on says, a request is not retried when:

  • the provider sent an x-should-retry: false header, as OpenAI and Anthropic do for errors that retrying won’t fix (see provider_retry_hint). Their SDKs heed the header too, so turning the SDK’s retries off doesn’t lose it. A ModelFallback can still move on to another model.
  • the run’s usage limits stopped it, as another attempt wouldn’t change that.

Once a model has been attempted max_attempts times (3 by default, the first attempt included), the last error is raised as is, or the last rejected response accepted, so that a ModelFallback can take over. The agent attempts a request at most 100 times across all its models, so beyond that a UserError is raised instead. Two RetryOnModelErrors capabilities, say one with its own retry_on for rate limits, count the same attempts, each against its own max_attempts.

Waiting between attempts

Before each retry, the agent waits as long as the error’s retry_after says the provider asked for. Without one, it backs off exponentially: initial_delay seconds (1 by default) before the first retry, doubling for each retry after it, up to max_delay seconds (30 by default). A random part of up to half of each wait is taken off, so that many runs failing at the same moment don’t all retry at the same moment too.

A request is not retried, and the error is raised at once, when the wait would be longer than max_delay, as a provider asking for minutes rather than seconds usually means a quota resets later, or when the wait would end after the request’s or the run’s deadline. Either way, a ModelFallback can then move on to another model while the run still has time.

A request_timeout covers a request to a model with all its retries and the waits between them: each retry gets what’s left of the deadline the first attempt started with, and a wait that would end after it isn’t made. A retry that runs out of time raises a ModelRequestTimeout whose timeout is what was left for that attempt, not the request_timeout you set. A model ModelFallback moves on to gets a fresh deadline, as without RetryOnModelErrors.

How attempts work

Each retry is an attempt within the same request step, as each model ModelFallback tries is. The model never sees a retry prompt, before_model_request and wrap_model_request run once per step, and prepare_model_request runs again for each attempt. See How attempts work for the details, which are the same.

usage.requests counts the step once, so UsageLimits.request_limit isn’t spent on retries. The tokens and cost of a response a response handler rejected are counted in the run’s usage and checked against its token and cost limits before the next attempt.

The response that answers lists every attempt before it in failed_attempts, retries and fallbacks in one list, in the order they were made. Each one’s kind says whether the same model was attempted after it ('retry') or the request moved on to another model ('fallback'), and its retry_after records the wait the provider asked for. When the agent is instrumented, each failed attempt also gets its own model request attempt span, as described under Failed attempts and telemetry.

Provider SDK retries

Most provider SDKs retry failed requests themselves before raising, so each attempt could otherwise be several requests on the wire, with waits of their own. RetryOnModelErrors turns those retries off for the requests it makes, by sending them with ModelSettings['max_retries'] set to 0, so it’s the one layer that retries and max_attempts is the number of requests sent. A max_retries you set yourself, on the agent, the run or a model, is kept, and a model that doesn’t support the setting isn’t sent it.

Streaming

A streamed request is retried when it fails before any of its output reaches you: the agent waits for the stream’s first event before handing the stream over, so a connection error, an error status, or an error the provider reports right after accepting the request can still be retried. Once output has been streamed, a failure is raised to you rather than retried. Response handlers aren’t consulted for a streamed response, which is accepted as is with a UserWarning. A streamed refusal isn’t retried either, without a warning, as it isn’t fallen back from. See Streaming for when a request is streamed.

In agent specs

RetryOnModelErrors can be declared in an agent spec. Its retry_on takes the names of ModelAPIError and its subclasses in pydantic_ai.exceptions, ContentFilterError and OutputTokenLimitExceeded; exception and response handlers can only be passed in Python:

retry_agent.yaml
model: openai:gpt-6.1-sol
capabilities:
  - RetryOnModelErrors:
      retry_on: [ModelRateLimitError, ModelOverloadedError]
      max_attempts: 5
  - ModelFallback: [anthropic:claude-sonnet-5-5]

Durable execution

Under a durability capability, each attempt is a durable model operation of its own, as under ModelFallback: on Temporal, an activity per attempt. The wait before a retry is a durable timer on Temporal and a durable sleep on DBOS, so a workflow that is recovered while it waits doesn’t wait again from the start. DBOS has no replay-safe clock for the run yet, though, so with a request_timeout or a run deadline, a workflow recovered partway through a model’s retries can decide differently whether to retry or fall back than it did the first time. On Prefect, the wait is an ordinary sleep in the flow.

On Temporal, an agent with RetryOnModelErrors makes every model error non-retryable in the model request activity, as if it were listed in TemporalDurability’s non_retryable_model_errors: the error fails the activity on its first attempt and reaches RetryOnModelErrors as its own class, so the capability alone retries it. A non_retryable_model_errors you pass other than the default is ignored then, with a UserWarning: choose which errors are retried with retry_on instead. Add RetryOnModelErrors to the agent’s capabilities for this, not through a DynamicCapability: the activity’s retry policy is set when the agent is bound, so one that only the run resolves raises a UserError inside a workflow. The activity’s retry policy still covers what never reaches a hook, like a worker crash or an activity timeout, so keep it, with a bounded maximum_attempts in model_activity_config, as Temporal’s default retries without limit. Don’t set maximum_attempts=1: a timed-out activity would then fail the workflow.

On DBOS and Prefect, the step’s own retries, off by default, retry a failed request before the error reaches RetryOnModelErrors, and the two multiply, so leave them off.

On AWS Lambda, the SDK retries every step six times by default, so with max_attempts=3 a model can be requested up to 18 times. Turn the step retries off with AWSLambdaDurability(step_config={'retry_strategy': RetryPresets.none()}) (RetryPresets is in aws_durable_execution_sdk_python.retries); a tool that should still be retried can set its own retry_strategy in its metadata. On Lambda, Absurd, pgtask, Azure Durable Task and Render Workflows, the wait before a retry is an ordinary sleep in the process, not a durable timer yet: a run that is recovered while it waits waits again, and on Lambda the wait is billed time.