Model Fallback
ModelFallback is a capability that attempts other models when the one serving a request fails, or returns a response you reject. It’s the way to fall back to other models in an agent; FallbackModel remains for direct model requests and standalone models (see Migrating from FallbackModel). The chain is the model selected for the step, followed by the capability’s models:
from pydantic_ai import Agent
from pydantic_ai.capabilities import ModelFallback
# Try the OpenAI model first, then the Anthropic one.
agent = Agent('openai:gpt-6.1-sol', capabilities=[ModelFallback('anthropic:claude-sonnet-5-5')])
Because the chain starts from the model selected for the step, ModelFallback composes with the agent’s model, a model passed to agent.run(model=...), and SelectModel: whichever model they choose is attempted first. When the agent has no model and nothing else selects one, the first of ModelFallback’s models is used for the first attempt:
from pydantic_ai import Agent
from pydantic_ai.capabilities import ModelFallback
agent = Agent(capabilities=[ModelFallback('openai:gpt-6.1-sol', 'anthropic:claude-sonnet-5-5')])
A model that was already attempted in the step is skipped, whichever capability attempted it, so listing the agent’s own model among the candidates doesn’t cause a duplicate request.
Falling back is not retrying: the next model usually can’t use the prompt cache that earlier requests built on the first one, and can answer differently and at a different price. Leave a transient failure, like a dropped connection, to a retry of the same model, with a budget small enough that it doesn’t hold up the fallback when the model stays down: add RetryOnModelErrors, and each model in the chain is retried before the request moves on to the next one (see Retry or fall back). See Recovery scopes compared and Choosing one retry owner.
By default, ModelFallback moves on when a request raises ModelAPIError, the error a model raises when the provider’s API fails, such as a 4xx or 5xx response, when the model refuses, and when the output is cut off by the token limit: fallback_on defaults to (ModelAPIError, ContentFilterError, OutputTokenLimitExceeded).
The fallback_on parameter takes the same forms as FallbackModel’s, whose default is (ModelAPIError,):
- a tuple of exception types
- an exception handler,
(Exception) -> bool - a response handler,
(ModelResponse) -> bool, which rejects a response that was returned successfully - a sequence mixing any of the above
Handlers may be sync or async. A handler whose first parameter is annotated as ModelResponse is a response handler; any other handler is an exception handler. See Response-Based Fallback for examples of handlers, which work the same for both.
Here the first model’s answer is blank and rejected by a response handler, the second model fails, and the third answers. The answering response lists both attempts it moved on from, and the rejected response’s tokens count in the run’s usage:
from pydantic_ai import Agent, ModelAPIError, ModelMessage, ModelResponse, TextPart
from pydantic_ai.capabilities import ModelFallback
from pydantic_ai.exceptions import ContentFilterError, OutputTokenLimitExceeded
from pydantic_ai.models.function import AgentInfo, FunctionModel
from pydantic_ai.usage import RequestUsage
def blank(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
return ModelResponse(
parts=[TextPart('\n\n')],
usage=RequestUsage(input_tokens=60, output_tokens=100),
)
def unavailable(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
raise ModelAPIError('second', 'Service unavailable')
def complete(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
return ModelResponse(
parts=[TextPart('The capital of France is Paris.')],
usage=RequestUsage(input_tokens=60, output_tokens=8),
)
def is_blank(response: ModelResponse) -> bool:
return not response.tool_calls and not (response.text or '').strip()
agent = Agent(
# These deterministic models stand in for three providers.
FunctionModel(blank, model_name='primary'),
capabilities=[
ModelFallback(
FunctionModel(unavailable, model_name='second'),
FunctionModel(complete, model_name='backup'),
fallback_on=[ModelAPIError, ContentFilterError, OutputTokenLimitExceeded, is_blank],
)
],
)
result = agent.run_sync('What is the capital of France?')
print(result.output)
#> The capital of France is Paris.
print(result.usage)
#> RunUsage(input_tokens=120, output_tokens=108, requests=1)
for attempt in result.response.failed_attempts or []:
print(attempt.model_name, attempt.outcome, attempt.usage)
#> primary rejected RequestUsage(input_tokens=60, output_tokens=100)
#> second error None
When every model has failed or been rejected, ModelFallback raises FallbackExceptionGroup with every exception, plus a ResponseRejected counting any rejected responses.
A run that reaches its timeout= stops with RunTimedOut wherever it is, including in the middle of a candidate’s request: the remaining candidates aren’t attempted. Likewise, a request that exceeds the run’s usage limits raises UsageLimitExceeded whatever fallback_on says, as another model wouldn’t change that.
A model sometimes refuses a request on safety grounds that another model would carry out, as a coding agent can find when a task touches on security. Pydantic AI reports a refusal as a successful response with a finish_reason of 'content_filter', and puts the provider’s explanation, where it gives one, in [provider_details['refusal']][pydantic_ai.messages.ModelResponse.provider_details]: Anthropic’s stop_reason: 'refusal' with its stop_details, the refusal that OpenAI sends in place of content, and Azure’s content filter.
ModelFallback moves on from a refusal by default, to the next model, which applies its own provider’s safety policy. That’s the capability doing what you add it for; the agent on its own keeps its content-filter behavior, so without ModelFallback an empty refusal raises ContentFilterError and one with text is the run’s output. It does so whether the response is empty or has partial text: Anthropic recommends treating any partial output of a refused response as incomplete, and pointing the retry at a fallback model rather than the same one, which usually refuses again. Naming ContentFilterError in fallback_on, as the default does, is what turns this on: the agent raises that error only after it has accepted a response, too late to fall back from it, so in fallback_on it stands for the response it’s raised for.
from pydantic_ai import Agent, ModelMessage, ModelResponse, TextPart
from pydantic_ai.capabilities import ModelFallback
from pydantic_ai.models.function import AgentInfo, FunctionModel
def refuse(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
return ModelResponse(
parts=[],
finish_reason='content_filter',
provider_details={'finish_reason': 'refusal', 'refusal': "I can't help with that request."},
)
def answer(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
return ModelResponse(
parts=[TextPart("The input isn't escaped before it reaches the query; here's the fix.")]
)
agent = Agent(
# Deterministic stand-ins: an Anthropic model that refuses, and an OpenAI model that takes over.
FunctionModel(refuse, model_name='anthropic-stand-in'),
capabilities=[ModelFallback(FunctionModel(answer, model_name='openai-stand-in'))],
)
result = agent.run_sync('Explain the vulnerability in this code and how to fix it.')
print(result.output)
#> The input isn't escaped before it reaches the query; here's the fix.
for attempt in result.response.failed_attempts or []:
print(attempt.model_name, attempt.outcome)
#> anthropic-stand-in rejected
A few things to know about falling back on refusals:
- To have a refusal end the run instead, opt out with
fallback_on=(ModelAPIError, OutputTokenLimitExceeded). An empty refusal then raisesContentFilterError, and one with text is the run’s output, as withoutModelFallback; addRaiseContentFilterErrorto raise on those too. - A
FallbackModeldoesn’t fall back on refusals by default. Pass itfallback_on=(ModelAPIError, ContentFilterError). - It happens per request. When a model refuses partway through a run, only that request moves to the next model, with the run’s message history so far. The responses the first model gave earlier in the run stay in the history. Anthropic recommends staying on the fallback model for the rest of the conversation, which
sticky=Truedoes. - Without a fallback, an empty refusal raises
ContentFilterError. It’s anUnexpectedModelBehavior, not aModelAPIError, and so is a thinking-only refusal. Its message starts withContent filter triggered.and then names the reason:Refusal: '...'with the text fromprovider_details['refusal']for an OpenAI refusal, orFinish reason: 'refusal'for an Anthropic one, whose refusal text is then only in the error’sbody, the serialized response.failed_attemptsrecords a refused attempt asrejected, without itsprovider_details. - A streamed refusal isn’t fallen back from. It’s only known once it has been streamed, so it’s accepted, without a warning, and an empty one then raises
ContentFilterError. To fall back there, catch it and run again with another model passed asmodel=. - Some refusals are caused by the prompt. Anthropic’s
reasoning_extractionrefusals come from prompts or tool schemas that ask the model to put its reasoning in its output, which Anthropic says to change instead (see Refusals on repeated prompts). Falling back to another provider still works for them.
A response that reaches the max_tokens model setting, or the model’s own output limit, is cut off, with a finish_reason of 'length'. Models differ in how much they write for the same request, so another one may answer within the limit. ModelFallback moves on from a cut-off response by default, because without it a cut-off text answer is the run’s output and looks complete. Naming OutputTokenLimitExceeded in fallback_on, as the default does, is what turns this on, as ContentFilterError does for refusals.
from pydantic_ai import Agent, ModelMessage, ModelResponse, TextPart
from pydantic_ai.capabilities import ModelFallback
from pydantic_ai.models.function import AgentInfo, FunctionModel
def cut_off(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
return ModelResponse(parts=[TextPart('The three largest cities are Tokyo, Delhi and')], finish_reason='length')
def answer(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
return ModelResponse(parts=[TextPart('Tokyo, Delhi and Shanghai.')])
agent = Agent(
# Deterministic stand-ins: a verbose model that runs out of tokens, and a terser one that takes over.
FunctionModel(cut_off, model_name='verbose-stand-in'),
capabilities=[ModelFallback(FunctionModel(answer, model_name='terse-stand-in'))],
)
result = agent.run_sync('Name the three largest cities.', model_settings={'max_tokens': 10})
print(result.output)
#> Tokyo, Delhi and Shanghai.
for attempt in result.response.failed_attempts or []:
print(attempt.model_name, attempt.outcome)
#> verbose-stand-in rejected
A few things to know about falling back on cut-off responses:
- To accept cut-off responses instead, opt out with
fallback_on=(ModelAPIError, ContentFilterError), for instance whenmax_tokensis a deliberate cap on the answer’s length, or you checkresult.response.finish_reasonyourself. Cut-off text is then the run’s output, as withoutModelFallback. - Every cut-off response falls back, whatever it holds: text, which the agent can’t tell from a complete answer, structured output or tool call arguments that may fail validation, and the rare one whose output is complete but still reached the limit, such as tool calls that all parse. A response that reached the limit is one the model didn’t finish, the way a refusal’s partial output is one the provider says to discard.
- When every model is cut off, the run raises.
ModelFallbackraisesFallbackExceptionGroup, where without it the last cut-off text would have been the output. Raisemax_tokensif that happens often. - A
FallbackModeldoesn’t fall back on cut-off responses by default. NameOutputTokenLimitExceededin itsfallback_on. - A streamed cut-off response isn’t fallen back from. Its finish reason is only known once it has been streamed, so it’s accepted, without a warning, and the agent then handles it as it would without
ModelFallback. IncompleteToolCalldoesn’t stand for cut-off responses. It’s theOutputTokenLimitExceededthe agent raises for a tool call whose arguments were cut off; nameOutputTokenLimitExceededitself.
ModelFallback can be declared in an agent spec, with its models as a list of model names:
model: openai:gpt-6.1-sol
capabilities:
- ModelFallback: [anthropic:claude-sonnet-5-5]
As in Python, the spec’s model: is attempted first, and a spec without one uses the first of ModelFallback’s models for the first attempt.
To choose when to fall back, give the models and fallback_on as keyword arguments. In a spec, fallback_on takes the names of ModelAPIError and its subclasses in pydantic_ai.exceptions, such as ModelRateLimitError and ModelUnavailableError, ContentFilterError and OutputTokenLimitExceeded. Exception and response handlers can only be passed in Python:
model: openai:gpt-6.1-sol
capabilities:
- ModelFallback:
models: [anthropic:claude-sonnet-5-5, google:gemini-3.8-flash]
fallback_on: [ModelRateLimitError, ModelUnavailableError]
A spec can also set sticky: true the same way.
Model names are resolved when they’re attempted, like those passed to ModelFallback(...), so a spec can also list model IDs of your own that a ResolveModelId capability resolves, passed to Agent.from_spec() or Agent.from_file() with capabilities=.
Each model attempted is an attempt within the same request step, not a new step:
before_model_requestandwrap_model_requestrun once per step, and the model never sees a retry prompt.prepare_model_requestruns again for every attempt, withrequest_context.modelset to the model about to serve it.RunContext.modelstays the model selected for the step, so a hook that depends on the serving model, its profile or its context window readsrequest_context.model. The model’s own message preparation and token counting are redone for each candidate, and so is any preparation a capability does inprepare_model_request, rather than inherited from the previous one. The built-in compaction capabilities, including the Harness ones, still run inbefore_model_requestfor now, so they decide once per step for the step’s model, and every candidate inherits that decision. See Retrying and falling back.- Instructions and model settings are resolved again for each model after the first, with that model as
ctx.model: the agent’s and capabilities’ instructions, includingget_instructionsfunctions, andmodel_settingsandget_model_settingscallables. So each model is sent the instructions and settings it would be sent on its own, and the request recorded in message history has the instructions and settings of the model that answered. Settings abefore_model_requesthook changed still apply on top. Instruction parts abefore_model_requesthook changed are sent to every model as the hook left them. Other hooks, and the run’s tools, see the step’s model asctx.model. See Instructions for each model. - A native tool factory is called again for each model after the first, with that model as
ctx.model, so each model is sent the native tools it would be sent on its own, and a provider-adaptive tool’s local fallback is offered to a model its native tool doesn’t apply to. A native tool abefore_model_requesthook removed stays removed. after_model_requesthooks of capabilities outsideModelFallbackonly ever see the response that was accepted.ModelFallbackis positioned innermost, so it judges a response before them.usage.requestscounts the step once, soUsageLimits.request_limitbounds how many steps the agent takes rather than how many models it tried. The tokens and cost of every attempt that produced a response, including rejected ones, are counted in the run’s usage and checked against token and cost limits before the next attempt.
Instructions and model settings that depend on the model can read it from ctx.model. ModelFallback resolves them again for each model it moves on to, and with sticky=True for the model a conversation stays on. That way, a model a request falls back to is sent its own instructions and settings, not the first model’s:
from dataclasses import dataclass
from typing import Any
from pydantic_ai import Agent, ModelMessage, ModelRequest, ModelResponse, RunContext, TextPart
from pydantic_ai.capabilities import AbstractCapability, ModelFallback
from pydantic_ai.exceptions import ModelAPIError
from pydantic_ai.models.function import AgentInfo, FunctionModel
from pydantic_ai.settings import ModelSettings
@dataclass
class PerModelTuning(AbstractCapability[Any]):
def get_instructions(self):
def instructions(ctx: RunContext[Any]) -> str:
if ctx.model.model_name == 'backup':
return 'Answer in one word.'
return 'Answer briefly.'
return instructions
def get_model_settings(self):
def settings(ctx: RunContext[Any]) -> ModelSettings:
return {'temperature': 0.0 if ctx.model.model_name == 'backup' else 0.7}
return settings
def primary_response(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
raise ModelAPIError(model_name='primary', message='Overloaded')
def backup_response(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
temperature = (info.model_settings or {}).get('temperature')
return ModelResponse(parts=[TextPart(f'{info.instructions} (temperature {temperature})')])
# These deterministic responses stand in for calls to a primary model that's down and a backup model.
primary = FunctionModel(primary_response, model_name='primary')
backup = FunctionModel(backup_response, model_name='backup')
agent = Agent(primary, capabilities=[PerModelTuning(), ModelFallback(backup)])
result = agent.run_sync('What is the capital of France?')
print(result.output)
#> Answer in one word. (temperature 0.0)
request = result.all_messages()[0]
assert isinstance(request, ModelRequest)
print(request.instructions)
#> Answer in one word.
Text that depends on the model is best left with the default on_change='rewrite': switching models starts a new prompt cache anyway, so rewriting the instructions costs nothing extra.
Only the instructions and model settings are resolved again: a toolset from get_toolset, before_model_request and wrap_model_request hooks, and the tools the model calls see the model selected for the step as ctx.model. Toolsets’ instructions, like an MCP server’s, are resolved once per step too, so an attempt doesn’t ask a toolset for them again under durable execution. The instructions of deferred capabilities the model has loaded, which are delivered through message history, are also resolved once per step. An agent or capability instruction function that makes a durable call runs again for each model after the first, and so makes that call again. A suspended response that’s continued keeps the instructions it was sent.
Failed attempts are recorded the same way FallbackModel records them. The response that answers lists every attempt ModelFallback moved on from in failed_attempts, along with any attempt RetryOnModelErrors retried, which has a kind of 'retry' rather than 'fallback', and when every model fails, the FallbackExceptionGroup lists them in attempts. A rejected response’s tokens and cost are counted once, in the run’s RunUsage, whichever of the two you use.
When the agent is instrumented, the model request (chat) span is named after the model that answered and describes it, with its usage. Each failed attempt gets its own model request attempt <model name> child span with an ERROR status: an error is recorded on it as an exception, and a rejected response carries its usage, cost and finish reason. The span names the model it tried in its gen_ai.request.model and gen_ai.provider.name attributes, and its zero-based position among the request’s attempts in pydantic_ai.model_request.attempt. When every model fails, the chat span is named after the last model attempted. See Failed attempts and usage for how these spans relate to the ones the tried model opens itself.
A streamed request falls back if it fails before any output reaches you: ModelFallback waits for each candidate’s stream to produce its first event before handing it over, so a connection error, an error status, or an error the provider reports right after accepting the request (like an overload) can still be handled. The first event is then replayed, so you see the same stream. Once output has been streamed, a failure is raised to you rather than retried. FallbackModel streams the same way.
Response handlers aren’t consulted for a streamed response, which is accepted as is with a UserWarning: it has already been streamed by the time it could be judged. A streamed refusal or cut-off response is accepted for the same reason, without a warning, as it would be without ModelFallback. A request is streamed whenever something consumes the run’s events: agent.run_stream(), agent.run_stream_events() (which the UI adapters use), agent.iter() with node.stream(), and agent.run() or agent.run_sync() with an event_stream_handler, an agent.on_event listener, or a capability that reacts to events. A capability of your own that raises RetryModelRequest from after_model_request on a streamed request gets a UserError.
By default, every request step starts the chain from the model selected for it, so a conversation that fell back goes back to the first model at its next step. With sticky=True, it keeps using the candidate that answered instead. That keeps hitting the same provider’s prompt cache, and doesn’t make every step wait on a model that’s failing:
from pydantic_ai import Agent, ModelMessage, ModelResponse, TextPart
from pydantic_ai.capabilities import ModelFallback
from pydantic_ai.exceptions import ModelAPIError
from pydantic_ai.models.function import AgentInfo, FunctionModel
def primary_response(messages: list[ModelMessage], _info: AgentInfo) -> ModelResponse:
raise ModelAPIError(model_name='primary', message='Overloaded')
def backup_response(messages: list[ModelMessage], _info: AgentInfo) -> ModelResponse:
return ModelResponse(parts=[TextPart('Paris')])
# These deterministic responses stand in for calls to a primary model that's down and a backup model.
primary = FunctionModel(primary_response, model_name='primary')
backup = FunctionModel(backup_response, model_name='backup')
agent = Agent(primary, capabilities=[ModelFallback(backup, sticky=True)])
result = agent.run_sync('What is the capital of France?')
print([attempt.model_name for attempt in result.response.failed_attempts or []])
#> ['primary']
result = agent.run_sync('Are you sure?', message_history=result.all_messages())
print(result.response.model_name, result.response.failed_attempts)
#> backup None
The second run goes straight to the backup model. Every response records the model that served it as model_id, and the attempts that failed before it as failed_attempts, so this holds for any later run that continues the conversation, in this process or another. Only a fallback makes a conversation sticky: an answer from a candidate you picked for one run, with agent.run(model=...), doesn’t change which model later runs start from. The conversation stays on the candidate it fell back to until it fails. The chain then moves on to the models after it, and comes back around to the step’s own model and the models before it, so no model the chain would have attempted is left out. Once the step’s own model answers, the conversation starts from it again.
sticky only replaces the agent’s default model: a model passed to agent.run(model=...) or Agent.override, or selected by a capability such as SelectModel, is still attempted first. A recorded candidate that’s no longer among ModelFallback’s models is ignored.
Some providers suspend a response and continue it on a later request, like Anthropic’s pause_turn or OpenAI’s background mode. Every response records the model that served it as model_id, so when a later run resumes a suspended response one of ModelFallback’s models started, the continuation goes to that model rather than to the first model in the chain. A suspended response started by a model outside the chain is continued by the step’s model, as without ModelFallback. Response handlers judge the continued response like any other. If that continuation fails or is rejected, the suspended response is dropped and the turn is generated again by the next model in the chain after the one it was pinned to, as it would have been had the response failed in the run that started it. The chain doesn’t start over: with batching, the first model would only accept a new batch, and the request would never reach the models after it. A rejected response the provider is still working on, like a batch request that hasn’t finished, has its job cancelled.
A request_timeout in the model settings bounds each model’s attempt as a whole, across the provider SDK’s own retries: each model ModelFallback tries gets a fresh deadline, and the ModelRequestTimeout it raises when the deadline passes moves on to the next model by default. A model’s own settings can set a different request_timeout or first_event_timeout for it, which applies only when neither the agent nor the run sets one, as those take precedence.
On a streamed request, first_event_timeout also bounds each model’s wait for its first output, and whichever deadline passes first applies: a model that hasn’t started streaming in time raises ModelRequestTimeout too, and ModelFallback moves on to the next model, which gets a fresh deadline. Once a response has started, the setting no longer applies, so a long response isn’t cut off.
With BatchModelRequests, every attempt is batched with the model that serves it, so when the first model’s batch request fails, the request goes into a batch on the next model. Batching submits a request again itself after a transient failure, so ModelFallback only sees a batch failure once resubmitting is used up, or one resubmitting wouldn’t fix.
Under a durability capability, each attempt is a durable model operation of its own: on Temporal, an activity per model tried. The activity’s retry policy retries a failed request before the error reaches ModelFallback’s fallback_on, and Temporal’s default retry policy retries without limit: set maximum_attempts in the retry policy of TemporalDurability’s model_activity_config so that a failing model doesn’t hold up the fallback. Each model’s request_timeout deadline runs as a durable timer around its own activity. Its first_event_timeout deadline runs inside the activity, which is what sees the stream’s events, so the ModelRequestTimeout it raises fails the activity there. ModelRequestTimeout is in non_retryable_model_errors by default, so the activity isn’t retried and ModelFallback moves on to the next model straight away. So are a rejected request and an exhausted quota.
A streamed request is consumed inside its durable unit, where its events reach a durability capability’s event_stream_handler, so a stream that fails after producing output there is raised rather than fallen back from, as outside durable execution, and what it produced counts towards the run’s usage.
A model ModelFallback moves to is rebuilt for its activity like any model a run uses: a model name string is built on the worker, and a model instance must be registered with the durability capability’s models, then given to ModelFallback by that name:
from pydantic_ai import Agent
from pydantic_ai.capabilities import ModelFallback
from pydantic_ai.durable_exec.temporal import TemporalDurability
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.providers.anthropic import AnthropicProvider
backup = AnthropicModel('claude-sonnet-5-5', provider=AnthropicProvider(api_key='...'))
agent = Agent(
'openai:gpt-6.1-sol',
name='resilient_agent',
capabilities=[
ModelFallback('backup'),
TemporalDurability(models={'backup': backup}),
],
)
Instructions and model settings are resolved again for each model a step falls back to (see Instructions for each model):
A FallbackModel instead runs its whole chain inside one activity, along with the deadlines it gives each model, and when every model fails, its FallbackExceptionGroup reaches the workflow without the attempts it lists, so the usage of the responses it rejected isn’t recorded.
ModelFallback is the recommended way to fall back to other models inside an agent. FallbackModel isn’t deprecated and doesn’t warn: like InstrumentedModel next to the Instrumentation capability, it remains the way to fall back with direct model requests or a standalone Model, outside an agent.
To move an agent to the capability, keep the first model as the agent’s model and pass the rest, with the same fallback_on, to ModelFallback:
# Before
agent = Agent(FallbackModel(primary, backup, fallback_on=[ModelAPIError, is_blank]))
# After
agent = Agent(primary, capabilities=[ModelFallback(backup, fallback_on=[ModelAPIError, is_blank])])
Everything a FallbackModel does carries over:
fallback_ontakes the same exception types, exception handlers and response handlers, mixed in a list, sync or async.- Each model keeps its own
settings, and settings on the agent or run apply to every model. - The answering response lists the attempts moved on from in
failed_attempts, and when every model fails,FallbackExceptionGroup.attemptslists them. - A rejected response’s tokens and cost count in the run’s usage and towards its limits, also when the run is cancelled, times out or ends on an error
fallback_ondoesn’t cover. - Each model gets fresh
request_timeoutandfirst_event_timeoutdeadlines. - A stream that fails before any output falls back, and response handlers don’t apply to a streamed response.
- A suspended response is continued on the model that started it.
- The same spans: a
chatspan for the model that answered, and amodel request attemptspan for each attempt moved on from.
Behavior differences:
- Refusals and cut-off responses fall back by default.
ModelFallback’s defaultfallback_onis(ModelAPIError, ContentFilterError, OutputTokenLimitExceeded), so it moves on when a model refuses or its output is cut off;FallbackModel’s is(ModelAPIError,). Passfallback_on=(ModelAPIError,)to keepFallbackModel’s behavior.
And a few things only the capability can do:
prepare_model_requestruns for each model, so hooks see the model that will serve the request, its profile and context window, and prepare the request for it. The model’s own message preparation and token counting are redone for each model too, socount_tokens_before_requestworks.- A native tool factory sees the model each attempt is at as
ctx.model, where under aFallbackModelit sees theFallbackModelitself. - The chain starts from the model selected for the step, so it composes with
agent.run(model=...)andSelectModel. - Under durable execution, each attempt is a durable model operation of its own.
- With batching, each attempt goes into a batch on its own model.
- A failed or rejected continuation of a resumed suspended response moves on to the next model rather than starting the chain over.