Skip to content

Model Fallback

ModelFallback is a capability that attempts other models when the one serving a request fails, or returns a response you reject. It’s the way to fall back to other models in an agent; FallbackModel remains for direct model requests and standalone models (see Migrating from FallbackModel). The chain is the model selected for the step, followed by the capability’s models:

model_fallback_capability.py
from pydantic_ai import Agent
from pydantic_ai.capabilities import ModelFallback

# Try the OpenAI model first, then the Anthropic one.
agent = Agent('openai:gpt-6.1-sol', capabilities=[ModelFallback('anthropic:claude-sonnet-5-5')])

Because the chain starts from the model selected for the step, ModelFallback composes with the agent’s model, a model passed to agent.run(model=...), and SelectModel: whichever model they choose is attempted first. When the agent has no model and nothing else selects one, the first of ModelFallback’s models is used for the first attempt:

model_fallback_without_agent_model.py
from pydantic_ai import Agent
from pydantic_ai.capabilities import ModelFallback

agent = Agent(capabilities=[ModelFallback('openai:gpt-6.1-sol', 'anthropic:claude-sonnet-5-5')])

A model that was already attempted in the step is skipped, whichever capability attempted it, so listing the agent’s own model among the candidates doesn’t cause a duplicate request.

Falling back is not retrying: the next model usually can’t use the prompt cache that earlier requests built on the first one, and can answer differently and at a different price. Leave a transient failure, like a dropped connection, to a retry of the same model, with a budget small enough that it doesn’t hold up the fallback when the model stays down: add RetryOnModelErrors, and each model in the chain is retried before the request moves on to the next one (see Retry or fall back). See Recovery scopes compared and Choosing one retry owner.

When to fall back

By default, ModelFallback moves on when a request raises ModelAPIError, the error a model raises when the provider’s API fails, such as a 4xx or 5xx response, when the model refuses, and when the output is cut off by the token limit: fallback_on defaults to (ModelAPIError, ContentFilterError, OutputTokenLimitExceeded).

The fallback_on parameter takes the same forms as FallbackModel’s, whose default is (ModelAPIError,):

  • a tuple of exception types
  • an exception handler, (Exception) -> bool
  • a response handler, (ModelResponse) -> bool, which rejects a response that was returned successfully
  • a sequence mixing any of the above

Handlers may be sync or async. A handler whose first parameter is annotated as ModelResponse is a response handler; any other handler is an exception handler. See Response-Based Fallback for examples of handlers, which work the same for both.

Here the first model’s answer is blank and rejected by a response handler, the second model fails, and the third answers. The answering response lists both attempts it moved on from, and the rejected response’s tokens count in the run’s usage:

model_fallback_mixed_policy.py
from pydantic_ai import Agent, ModelAPIError, ModelMessage, ModelResponse, TextPart
from pydantic_ai.capabilities import ModelFallback
from pydantic_ai.exceptions import ContentFilterError, OutputTokenLimitExceeded
from pydantic_ai.models.function import AgentInfo, FunctionModel
from pydantic_ai.usage import RequestUsage


def blank(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
    return ModelResponse(
        parts=[TextPart('\n\n')],
        usage=RequestUsage(input_tokens=60, output_tokens=100),
    )


def unavailable(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
    raise ModelAPIError('second', 'Service unavailable')


def complete(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
    return ModelResponse(
        parts=[TextPart('The capital of France is Paris.')],
        usage=RequestUsage(input_tokens=60, output_tokens=8),
    )


def is_blank(response: ModelResponse) -> bool:
    return not response.tool_calls and not (response.text or '').strip()


agent = Agent(
    # These deterministic models stand in for three providers.
    FunctionModel(blank, model_name='primary'),
    capabilities=[
        ModelFallback(
            FunctionModel(unavailable, model_name='second'),
            FunctionModel(complete, model_name='backup'),
            fallback_on=[ModelAPIError, ContentFilterError, OutputTokenLimitExceeded, is_blank],
        )
    ],
)
result = agent.run_sync('What is the capital of France?')
print(result.output)
#> The capital of France is Paris.
print(result.usage)
#> RunUsage(input_tokens=120, output_tokens=108, requests=1)
for attempt in result.response.failed_attempts or []:
    print(attempt.model_name, attempt.outcome, attempt.usage)
    #> primary rejected RequestUsage(input_tokens=60, output_tokens=100)
    #> second error None

When every model has failed or been rejected, ModelFallback raises FallbackExceptionGroup with every exception, plus a ResponseRejected counting any rejected responses.

A run that reaches its timeout= stops with RunTimedOut wherever it is, including in the middle of a candidate’s request: the remaining candidates aren’t attempted. Likewise, a request that exceeds the run’s usage limits raises UsageLimitExceeded whatever fallback_on says, as another model wouldn’t change that.

Falling back when a model refuses

A model sometimes refuses a request on safety grounds that another model would carry out, as a coding agent can find when a task touches on security. Pydantic AI reports a refusal as a successful response with a finish_reason of 'content_filter', and puts the provider’s explanation, where it gives one, in [provider_details['refusal']][pydantic_ai.messages.ModelResponse.provider_details]: Anthropic’s stop_reason: 'refusal' with its stop_details, the refusal that OpenAI sends in place of content, and Azure’s content filter.

ModelFallback moves on from a refusal by default, to the next model, which applies its own provider’s safety policy. That’s the capability doing what you add it for; the agent on its own keeps its content-filter behavior, so without ModelFallback an empty refusal raises ContentFilterError and one with text is the run’s output. It does so whether the response is empty or has partial text: Anthropic recommends treating any partial output of a refused response as incomplete, and pointing the retry at a fallback model rather than the same one, which usually refuses again. Naming ContentFilterError in fallback_on, as the default does, is what turns this on: the agent raises that error only after it has accepted a response, too late to fall back from it, so in fallback_on it stands for the response it’s raised for.

model_fallback_on_refusal.py
from pydantic_ai import Agent, ModelMessage, ModelResponse, TextPart
from pydantic_ai.capabilities import ModelFallback
from pydantic_ai.models.function import AgentInfo, FunctionModel


def refuse(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
    return ModelResponse(
        parts=[],
        finish_reason='content_filter',
        provider_details={'finish_reason': 'refusal', 'refusal': "I can't help with that request."},
    )


def answer(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
    return ModelResponse(
        parts=[TextPart("The input isn't escaped before it reaches the query; here's the fix.")]
    )


agent = Agent(
    # Deterministic stand-ins: an Anthropic model that refuses, and an OpenAI model that takes over.
    FunctionModel(refuse, model_name='anthropic-stand-in'),
    capabilities=[ModelFallback(FunctionModel(answer, model_name='openai-stand-in'))],
)
result = agent.run_sync('Explain the vulnerability in this code and how to fix it.')
print(result.output)
#> The input isn't escaped before it reaches the query; here's the fix.
for attempt in result.response.failed_attempts or []:
    print(attempt.model_name, attempt.outcome)
    #> anthropic-stand-in rejected

A few things to know about falling back on refusals:

  • To have a refusal end the run instead, opt out with fallback_on=(ModelAPIError, OutputTokenLimitExceeded). An empty refusal then raises ContentFilterError, and one with text is the run’s output, as without ModelFallback; add RaiseContentFilterError to raise on those too.
  • A FallbackModel doesn’t fall back on refusals by default. Pass it fallback_on=(ModelAPIError, ContentFilterError).
  • It happens per request. When a model refuses partway through a run, only that request moves to the next model, with the run’s message history so far. The responses the first model gave earlier in the run stay in the history. Anthropic recommends staying on the fallback model for the rest of the conversation, which sticky=True does.
  • Without a fallback, an empty refusal raises ContentFilterError. It’s an UnexpectedModelBehavior, not a ModelAPIError, and so is a thinking-only refusal. Its message starts with Content filter triggered. and then names the reason: Refusal: '...' with the text from provider_details['refusal'] for an OpenAI refusal, or Finish reason: 'refusal' for an Anthropic one, whose refusal text is then only in the error’s body, the serialized response. failed_attempts records a refused attempt as rejected, without its provider_details.
  • A streamed refusal isn’t fallen back from. It’s only known once it has been streamed, so it’s accepted, without a warning, and an empty one then raises ContentFilterError. To fall back there, catch it and run again with another model passed as model=.
  • Some refusals are caused by the prompt. Anthropic’s reasoning_extraction refusals come from prompts or tool schemas that ask the model to put its reasoning in its output, which Anthropic says to change instead (see Refusals on repeated prompts). Falling back to another provider still works for them.

Falling back when the output is cut off

A response that reaches the max_tokens model setting, or the model’s own output limit, is cut off, with a finish_reason of 'length'. Models differ in how much they write for the same request, so another one may answer within the limit. ModelFallback moves on from a cut-off response by default, because without it a cut-off text answer is the run’s output and looks complete. Naming OutputTokenLimitExceeded in fallback_on, as the default does, is what turns this on, as ContentFilterError does for refusals.

model_fallback_on_token_limit.py
from pydantic_ai import Agent, ModelMessage, ModelResponse, TextPart
from pydantic_ai.capabilities import ModelFallback
from pydantic_ai.models.function import AgentInfo, FunctionModel


def cut_off(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
    return ModelResponse(parts=[TextPart('The three largest cities are Tokyo, Delhi and')], finish_reason='length')


def answer(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
    return ModelResponse(parts=[TextPart('Tokyo, Delhi and Shanghai.')])


agent = Agent(
    # Deterministic stand-ins: a verbose model that runs out of tokens, and a terser one that takes over.
    FunctionModel(cut_off, model_name='verbose-stand-in'),
    capabilities=[ModelFallback(FunctionModel(answer, model_name='terse-stand-in'))],
)
result = agent.run_sync('Name the three largest cities.', model_settings={'max_tokens': 10})
print(result.output)
#> Tokyo, Delhi and Shanghai.
for attempt in result.response.failed_attempts or []:
    print(attempt.model_name, attempt.outcome)
    #> verbose-stand-in rejected

A few things to know about falling back on cut-off responses:

  • To accept cut-off responses instead, opt out with fallback_on=(ModelAPIError, ContentFilterError), for instance when max_tokens is a deliberate cap on the answer’s length, or you check result.response.finish_reason yourself. Cut-off text is then the run’s output, as without ModelFallback.
  • Every cut-off response falls back, whatever it holds: text, which the agent can’t tell from a complete answer, structured output or tool call arguments that may fail validation, and the rare one whose output is complete but still reached the limit, such as tool calls that all parse. A response that reached the limit is one the model didn’t finish, the way a refusal’s partial output is one the provider says to discard.
  • When every model is cut off, the run raises. ModelFallback raises FallbackExceptionGroup, where without it the last cut-off text would have been the output. Raise max_tokens if that happens often.
  • A FallbackModel doesn’t fall back on cut-off responses by default. Name OutputTokenLimitExceeded in its fallback_on.
  • A streamed cut-off response isn’t fallen back from. Its finish reason is only known once it has been streamed, so it’s accepted, without a warning, and the agent then handles it as it would without ModelFallback.
  • IncompleteToolCall doesn’t stand for cut-off responses. It’s the OutputTokenLimitExceeded the agent raises for a tool call whose arguments were cut off; name OutputTokenLimitExceeded itself.

In agent specs

ModelFallback can be declared in an agent spec, with its models as a list of model names:

fallback_agent.yaml
model: openai:gpt-6.1-sol
capabilities:
  - ModelFallback: [anthropic:claude-sonnet-5-5]

As in Python, the spec’s model: is attempted first, and a spec without one uses the first of ModelFallback’s models for the first attempt.

To choose when to fall back, give the models and fallback_on as keyword arguments. In a spec, fallback_on takes the names of ModelAPIError and its subclasses in pydantic_ai.exceptions, such as ModelRateLimitError and ModelUnavailableError, ContentFilterError and OutputTokenLimitExceeded. Exception and response handlers can only be passed in Python:

fallback_on_agent.yaml
model: openai:gpt-6.1-sol
capabilities:
  - ModelFallback:
      models: [anthropic:claude-sonnet-5-5, google:gemini-3.8-flash]
      fallback_on: [ModelRateLimitError, ModelUnavailableError]

A spec can also set sticky: true the same way.

Model names are resolved when they’re attempted, like those passed to ModelFallback(...), so a spec can also list model IDs of your own that a ResolveModelId capability resolves, passed to Agent.from_spec() or Agent.from_file() with capabilities=.

How attempts work

Each model attempted is an attempt within the same request step, not a new step:

  • before_model_request and wrap_model_request run once per step, and the model never sees a retry prompt.
  • prepare_model_request runs again for every attempt, with request_context.model set to the model about to serve it. RunContext.model stays the model selected for the step, so a hook that depends on the serving model, its profile or its context window reads request_context.model. The model’s own message preparation and token counting are redone for each candidate, and so is any preparation a capability does in prepare_model_request, rather than inherited from the previous one. The built-in compaction capabilities, including the Harness ones, still run in before_model_request for now, so they decide once per step for the step’s model, and every candidate inherits that decision. See Retrying and falling back.
  • Instructions and model settings are resolved again for each model after the first, with that model as ctx.model: the agent’s and capabilities’ instructions, including get_instructions functions, and model_settings and get_model_settings callables. So each model is sent the instructions and settings it would be sent on its own, and the request recorded in message history has the instructions and settings of the model that answered. Settings a before_model_request hook changed still apply on top. Instruction parts a before_model_request hook changed are sent to every model as the hook left them. Other hooks, and the run’s tools, see the step’s model as ctx.model. See Instructions for each model.
  • A native tool factory is called again for each model after the first, with that model as ctx.model, so each model is sent the native tools it would be sent on its own, and a provider-adaptive tool’s local fallback is offered to a model its native tool doesn’t apply to. A native tool a before_model_request hook removed stays removed.
  • after_model_request hooks of capabilities outside ModelFallback only ever see the response that was accepted. ModelFallback is positioned innermost, so it judges a response before them.
  • usage.requests counts the step once, so UsageLimits.request_limit bounds how many steps the agent takes rather than how many models it tried. The tokens and cost of every attempt that produced a response, including rejected ones, are counted in the run’s usage and checked against token and cost limits before the next attempt.

Instructions for each model

Instructions and model settings that depend on the model can read it from ctx.model. ModelFallback resolves them again for each model it moves on to, and with sticky=True for the model a conversation stays on. That way, a model a request falls back to is sent its own instructions and settings, not the first model’s:

model_fallback_instructions.py
from dataclasses import dataclass
from typing import Any

from pydantic_ai import Agent, ModelMessage, ModelRequest, ModelResponse, RunContext, TextPart
from pydantic_ai.capabilities import AbstractCapability, ModelFallback
from pydantic_ai.exceptions import ModelAPIError
from pydantic_ai.models.function import AgentInfo, FunctionModel
from pydantic_ai.settings import ModelSettings


@dataclass
class PerModelTuning(AbstractCapability[Any]):
    def get_instructions(self):
        def instructions(ctx: RunContext[Any]) -> str:
            if ctx.model.model_name == 'backup':
                return 'Answer in one word.'
            return 'Answer briefly.'

        return instructions

    def get_model_settings(self):
        def settings(ctx: RunContext[Any]) -> ModelSettings:
            return {'temperature': 0.0 if ctx.model.model_name == 'backup' else 0.7}

        return settings


def primary_response(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
    raise ModelAPIError(model_name='primary', message='Overloaded')


def backup_response(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
    temperature = (info.model_settings or {}).get('temperature')
    return ModelResponse(parts=[TextPart(f'{info.instructions} (temperature {temperature})')])


# These deterministic responses stand in for calls to a primary model that's down and a backup model.
primary = FunctionModel(primary_response, model_name='primary')
backup = FunctionModel(backup_response, model_name='backup')

agent = Agent(primary, capabilities=[PerModelTuning(), ModelFallback(backup)])

result = agent.run_sync('What is the capital of France?')
print(result.output)
#> Answer in one word. (temperature 0.0)
request = result.all_messages()[0]
assert isinstance(request, ModelRequest)
print(request.instructions)
#> Answer in one word.

Text that depends on the model is best left with the default on_change='rewrite': switching models starts a new prompt cache anyway, so rewriting the instructions costs nothing extra.

Only the instructions and model settings are resolved again: a toolset from get_toolset, before_model_request and wrap_model_request hooks, and the tools the model calls see the model selected for the step as ctx.model. Toolsets’ instructions, like an MCP server’s, are resolved once per step too, so an attempt doesn’t ask a toolset for them again under durable execution. The instructions of deferred capabilities the model has loaded, which are delivered through message history, are also resolved once per step. An agent or capability instruction function that makes a durable call runs again for each model after the first, and so makes that call again. A suspended response that’s continued keeps the instructions it was sent.

Failed attempts and telemetry

Failed attempts are recorded the same way FallbackModel records them. The response that answers lists every attempt ModelFallback moved on from in failed_attempts, along with any attempt RetryOnModelErrors retried, which has a kind of 'retry' rather than 'fallback', and when every model fails, the FallbackExceptionGroup lists them in attempts. A rejected response’s tokens and cost are counted once, in the run’s RunUsage, whichever of the two you use.

When the agent is instrumented, the model request (chat) span is named after the model that answered and describes it, with its usage. Each failed attempt gets its own model request attempt <model name> child span with an ERROR status: an error is recorded on it as an exception, and a rejected response carries its usage, cost and finish reason. The span names the model it tried in its gen_ai.request.model and gen_ai.provider.name attributes, and its zero-based position among the request’s attempts in pydantic_ai.model_request.attempt. When every model fails, the chat span is named after the last model attempted. See Failed attempts and usage for how these spans relate to the ones the tried model opens itself.

Streaming

A streamed request falls back if it fails before any output reaches you: ModelFallback waits for each candidate’s stream to produce its first event before handing it over, so a connection error, an error status, or an error the provider reports right after accepting the request (like an overload) can still be handled. The first event is then replayed, so you see the same stream. Once output has been streamed, a failure is raised to you rather than retried. FallbackModel streams the same way.

Response handlers aren’t consulted for a streamed response, which is accepted as is with a UserWarning: it has already been streamed by the time it could be judged. A streamed refusal or cut-off response is accepted for the same reason, without a warning, as it would be without ModelFallback. A request is streamed whenever something consumes the run’s events: agent.run_stream(), agent.run_stream_events() (which the UI adapters use), agent.iter() with node.stream(), and agent.run() or agent.run_sync() with an event_stream_handler, an agent.on_event listener, or a capability that reacts to events. A capability of your own that raises RetryModelRequest from after_model_request on a streamed request gets a UserError.

Staying on the model that answered

By default, every request step starts the chain from the model selected for it, so a conversation that fell back goes back to the first model at its next step. With sticky=True, it keeps using the candidate that answered instead. That keeps hitting the same provider’s prompt cache, and doesn’t make every step wait on a model that’s failing:

model_fallback_sticky.py
from pydantic_ai import Agent, ModelMessage, ModelResponse, TextPart
from pydantic_ai.capabilities import ModelFallback
from pydantic_ai.exceptions import ModelAPIError
from pydantic_ai.models.function import AgentInfo, FunctionModel


def primary_response(messages: list[ModelMessage], _info: AgentInfo) -> ModelResponse:
    raise ModelAPIError(model_name='primary', message='Overloaded')


def backup_response(messages: list[ModelMessage], _info: AgentInfo) -> ModelResponse:
    return ModelResponse(parts=[TextPart('Paris')])


# These deterministic responses stand in for calls to a primary model that's down and a backup model.
primary = FunctionModel(primary_response, model_name='primary')
backup = FunctionModel(backup_response, model_name='backup')

agent = Agent(primary, capabilities=[ModelFallback(backup, sticky=True)])

result = agent.run_sync('What is the capital of France?')
print([attempt.model_name for attempt in result.response.failed_attempts or []])
#> ['primary']
result = agent.run_sync('Are you sure?', message_history=result.all_messages())
print(result.response.model_name, result.response.failed_attempts)
#> backup None

The second run goes straight to the backup model. Every response records the model that served it as model_id, and the attempts that failed before it as failed_attempts, so this holds for any later run that continues the conversation, in this process or another. Only a fallback makes a conversation sticky: an answer from a candidate you picked for one run, with agent.run(model=...), doesn’t change which model later runs start from. The conversation stays on the candidate it fell back to until it fails. The chain then moves on to the models after it, and comes back around to the step’s own model and the models before it, so no model the chain would have attempted is left out. Once the step’s own model answers, the conversation starts from it again.

sticky only replaces the agent’s default model: a model passed to agent.run(model=...) or Agent.override, or selected by a capability such as SelectModel, is still attempted first. A recorded candidate that’s no longer among ModelFallback’s models is ignored.

Continuing suspended responses

Some providers suspend a response and continue it on a later request, like Anthropic’s pause_turn or OpenAI’s background mode. Every response records the model that served it as model_id, so when a later run resumes a suspended response one of ModelFallback’s models started, the continuation goes to that model rather than to the first model in the chain. A suspended response started by a model outside the chain is continued by the step’s model, as without ModelFallback. Response handlers judge the continued response like any other. If that continuation fails or is rejected, the suspended response is dropped and the turn is generated again by the next model in the chain after the one it was pinned to, as it would have been had the response failed in the run that started it. The chain doesn’t start over: with batching, the first model would only accept a new batch, and the request would never reach the models after it. A rejected response the provider is still working on, like a batch request that hasn’t finished, has its job cancelled.

Request deadlines

A request_timeout in the model settings bounds each model’s attempt as a whole, across the provider SDK’s own retries: each model ModelFallback tries gets a fresh deadline, and the ModelRequestTimeout it raises when the deadline passes moves on to the next model by default. A model’s own settings can set a different request_timeout or first_event_timeout for it, which applies only when neither the agent nor the run sets one, as those take precedence.

On a streamed request, first_event_timeout also bounds each model’s wait for its first output, and whichever deadline passes first applies: a model that hasn’t started streaming in time raises ModelRequestTimeout too, and ModelFallback moves on to the next model, which gets a fresh deadline. Once a response has started, the setting no longer applies, so a long response isn’t cut off.

Batching

With BatchModelRequests, every attempt is batched with the model that serves it, so when the first model’s batch request fails, the request goes into a batch on the next model. Batching submits a request again itself after a transient failure, so ModelFallback only sees a batch failure once resubmitting is used up, or one resubmitting wouldn’t fix.

Durable execution

Under a durability capability, each attempt is a durable model operation of its own: on Temporal, an activity per model tried. The activity’s retry policy retries a failed request before the error reaches ModelFallback’s fallback_on, and Temporal’s default retry policy retries without limit: set maximum_attempts in the retry policy of TemporalDurability’s model_activity_config so that a failing model doesn’t hold up the fallback. Each model’s request_timeout deadline runs as a durable timer around its own activity. Its first_event_timeout deadline runs inside the activity, which is what sees the stream’s events, so the ModelRequestTimeout it raises fails the activity there. ModelRequestTimeout is in non_retryable_model_errors by default, so the activity isn’t retried and ModelFallback moves on to the next model straight away. So are a rejected request and an exhausted quota.

A streamed request is consumed inside its durable unit, where its events reach a durability capability’s event_stream_handler, so a stream that fails after producing output there is raised rather than fallen back from, as outside durable execution, and what it produced counts towards the run’s usage.

A model ModelFallback moves to is rebuilt for its activity like any model a run uses: a model name string is built on the worker, and a model instance must be registered with the durability capability’s models, then given to ModelFallback by that name:

model_fallback_temporal.py
from pydantic_ai import Agent
from pydantic_ai.capabilities import ModelFallback
from pydantic_ai.durable_exec.temporal import TemporalDurability
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.providers.anthropic import AnthropicProvider

backup = AnthropicModel('claude-sonnet-5-5', provider=AnthropicProvider(api_key='...'))

agent = Agent(
    'openai:gpt-6.1-sol',
    name='resilient_agent',
    capabilities=[
        ModelFallback('backup'),
        TemporalDurability(models={'backup': backup}),
    ],
)

Instructions and model settings are resolved again for each model a step falls back to (see Instructions for each model):

A FallbackModel instead runs its whole chain inside one activity, along with the deadlines it gives each model, and when every model fails, its FallbackExceptionGroup reaches the workflow without the attempts it lists, so the usage of the responses it rejected isn’t recorded.

Migrating from FallbackModel

ModelFallback is the recommended way to fall back to other models inside an agent. FallbackModel isn’t deprecated and doesn’t warn: like InstrumentedModel next to the Instrumentation capability, it remains the way to fall back with direct model requests or a standalone Model, outside an agent.

To move an agent to the capability, keep the first model as the agent’s model and pass the rest, with the same fallback_on, to ModelFallback:

model_fallback_migration.py
# Before
agent = Agent(FallbackModel(primary, backup, fallback_on=[ModelAPIError, is_blank]))

# After
agent = Agent(primary, capabilities=[ModelFallback(backup, fallback_on=[ModelAPIError, is_blank])])

Everything a FallbackModel does carries over:

  • fallback_on takes the same exception types, exception handlers and response handlers, mixed in a list, sync or async.
  • Each model keeps its own settings, and settings on the agent or run apply to every model.
  • The answering response lists the attempts moved on from in failed_attempts, and when every model fails, FallbackExceptionGroup.attempts lists them.
  • A rejected response’s tokens and cost count in the run’s usage and towards its limits, also when the run is cancelled, times out or ends on an error fallback_on doesn’t cover.
  • Each model gets fresh request_timeout and first_event_timeout deadlines.
  • A stream that fails before any output falls back, and response handlers don’t apply to a streamed response.
  • A suspended response is continued on the model that started it.
  • The same spans: a chat span for the model that answered, and a model request attempt span for each attempt moved on from.

Behavior differences:

  • Refusals and cut-off responses fall back by default. ModelFallback’s default fallback_on is (ModelAPIError, ContentFilterError, OutputTokenLimitExceeded), so it moves on when a model refuses or its output is cut off; FallbackModel’s is (ModelAPIError,). Pass fallback_on=(ModelAPIError,) to keep FallbackModel’s behavior.

And a few things only the capability can do:

  • prepare_model_request runs for each model, so hooks see the model that will serve the request, its profile and context window, and prepare the request for it. The model’s own message preparation and token counting are redone for each model too, so count_tokens_before_request works.
  • A native tool factory sees the model each attempt is at as ctx.model, where under a FallbackModel it sees the FallbackModel itself.
  • The chain starts from the model selected for the step, so it composes with agent.run(model=...) and SelectModel.
  • Under durable execution, each attempt is a durable model operation of its own.
  • With batching, each attempt goes into a batch on its own model.
  • A failed or rejected continuation of a resumed suspended response moves on to the next model rather than starting the chain over.