pydantic_ai.providers
Bases: ABC, Generic[InterfaceClient]
Abstract class for a provider.
The provider is in charge of providing an authenticated client to the API.
Each provider only supports a specific interface. An interface can be supported by multiple providers.
For example, the OpenAIChatModel interface can be supported by the OpenAIProvider and the DeepSeekProvider.
When used as an async context manager, providers that create their own HTTP client will close it on exit.
This is handled automatically when using Agent as a context manager.
The provider name.
The returned value flows into ModelMessage.provider_name
on every part. Thinking-tag detection and native-tool detection check this value when
the model class loads history, so silently renaming a concrete name value breaks
replay of any message history captured against the old name.
Type: str
The namespace used to fully qualify model IDs routed through this provider.
Type: str
The base URL for the provider API.
Type: str
The client for the provider.
Type: InterfaceClient
@staticmethod
def model_profile(model_name: str) -> ModelProfile | None
The model profile for the named model, if available.
@staticmethod
def realtime_model_profile(model_name: str) -> RealtimeModelProfile | None
The realtime model profile for the named model, if available.
RealtimeModelProfile | None
def infer_provider(provider: str) -> Provider[Any]
Infer the provider from the provider name.
Provider[Any]
def infer_provider_class(provider: str) -> type[Provider[Any]]
Infers the provider class from the provider name.
def gateway_provider(
upstream_provider: Literal['openai', 'openai-chat', 'openai-responses', 'chat', 'responses'],
/,
*,
route: str | None = None,
api_key: str | None = None,
base_url: str | None = None,
http_client: AsyncHTTPClient | None = None,
) -> Provider[AsyncOpenAI]
def gateway_provider(
upstream_provider: Literal['groq'],
/,
*,
route: str | None = None,
api_key: str | None = None,
base_url: str | None = None,
http_client: httpx.AsyncClient | None = None,
) -> Provider[AsyncGroq]
def gateway_provider(
upstream_provider: Literal['anthropic'],
/,
*,
route: str | None = None,
api_key: str | None = None,
base_url: str | None = None,
http_client: httpx2.AsyncClient | None = None,
) -> Provider[AsyncAnthropicClient]
def gateway_provider(
upstream_provider: Literal['bedrock', 'converse'],
/,
*,
route: str | None = None,
api_key: str | None = None,
base_url: str | None = None,
) -> Provider[BaseClient]
def gateway_provider(
upstream_provider: Literal['google', 'google-cloud'],
/,
*,
route: str | None = None,
api_key: str | None = None,
base_url: str | None = None,
http_client: AsyncHTTPClient | None = None,
) -> Provider[GoogleClient]
def gateway_provider(
upstream_provider: str,
/,
*,
route: str | None = None,
api_key: str | None = None,
base_url: str | None = None,
) -> Provider[Any]
Create a new Gateway provider.
Provider[Any]
upstream_provider : UpstreamProvider | str
The upstream provider to use.
The name of the provider or gateway endpoint to use to handle the request. If not provided, the default gateway endpoint for the API format will be used.
The API key to use for authentication. If not provided, the PYDANTIC_AI_GATEWAY_API_KEY
environment variable will be used if available.
The base URL to use for the Gateway. If not provided, the PYDANTIC_AI_GATEWAY_BASE_URL
environment variable will be used if available. Otherwise, it is inferred from the API key’s
region, e.g. https://gateway-us.pydantic.dev/proxy.
http_client : AsyncHTTPClient | None Default: None
The HTTP client to use for the Gateway.
Bases: Provider[AsyncAnthropicClient]
Provider for Anthropic API.
def __init__(*, anthropic_client: AsyncAnthropicClient | None = None) -> None
def __init__(
*,
api_key: str | None = None,
base_url: str | None = None,
http_client: httpx2.AsyncClient | None = None,
) -> None
Create a new Anthropic provider.
The API key to use for authentication, if not provided, the ANTHROPIC_API_KEY environment variable
will be used if available.
The base URL to use for the Anthropic API.
anthropic_client : AsyncAnthropicClient | None Default: None
An existing Anthropic client to use. Accepts
AsyncAnthropic,
AsyncAnthropicBedrock,
AsyncAnthropicBedrockMantle,
AsyncAnthropicFoundry, or
AsyncAnthropicVertex.
If provided, the api_key and http_client arguments will be ignored.
http_client : httpx2.AsyncClient | None Default: None
An existing httpx2.AsyncClient to use for making HTTP requests.
Bases: Provider[Client], ABC
Common base for the Gemini API and Google Cloud providers.
Abstract — instantiate GoogleProvider for the
Gemini API or GoogleCloudProvider for
Google Cloud. Subclasses share base_url, client, _set_http_client, and model-profile
lookup; each subclass owns its own Client construction.
Bases: BaseGoogleProvider
Provider for the Gemini API (formerly Google AI Studio / Google GLA).
def __init__(
*,
api_key: str | None = None,
http_client: AsyncHTTPClient | None = None,
base_url: str | None = None,
retry_options: HttpRetryOptions | None = None,
) -> None
def __init__(*, client: Client) -> None
Create a new Google provider for the Gemini API.
The API key to
use for authentication. It can also be set via the GOOGLE_API_KEY environment variable,
or the legacy GEMINI_API_KEY environment variable (GOOGLE_API_KEY takes precedence).
client : Client | None Default: None
A pre-initialized client to use. Stored as-is, so a client built for Google Cloud
(Client(vertexai=True, ...), or its current spelling Client(enterprise=True, ...))
is used on that transport even though name stays 'google'. GoogleModel reads the
transport off the client rather than the provider name, so it routes correctly either way.
http_client : AsyncHTTPClient | None Default: None
An existing httpx2.AsyncClient or legacy httpx.AsyncClient to use for making HTTP requests.
The base URL for the Gemini API.
retry_options : HttpRetryOptions | None Default: None
HTTP retry options for transient errors (429, 5xx, etc.).
See google.genai.types.HttpRetryOptions for available fields.
Regions available for Google Cloud. More details here.
This lists single-region values only. GoogleCloudProvider also accepts the 'global' location and the
'us'/'eu' multi-regions (routed to the aiplatform.{us,eu}.rep.googleapis.com data-residency endpoints)
as separate union members on its location parameter.
Default: Literal['asia-east1', 'asia-east2', 'asia-northeast1', 'asia-northeast3', 'asia-south1', 'asia-southeast1', 'australia-southeast1', 'europe-central2', 'europe-north1', 'europe-southwest1', 'europe-west1', 'europe-west2', 'europe-west3', 'europe-west4', 'europe-west6', 'europe-west8', 'europe-west9', 'me-central1', 'me-central2', 'me-west1', 'northamerica-northeast1', 'southamerica-east1', 'us-central1', 'us-east1', 'us-east4', 'us-east5', 'us-south1', 'us-west1', 'us-west4']
Bases: BaseGoogleProvider
Provider for Google Cloud (formerly known as Vertex AI).
def __init__(
*,
api_key: str | None = None,
credentials: Credentials | None = None,
project: str | None = None,
location: GoogleCloudLocation | Literal['global', 'us', 'eu'] | str | None = None,
client: Client | None = None,
http_client: AsyncHTTPClient | None = None,
base_url: str | None = None,
retry_options: HttpRetryOptions | None = None,
) -> None
Create a new Google Cloud provider.
The Vertex AI Express Mode API key
to use for authentication. It can also be set via the GOOGLE_API_KEY environment variable,
or the legacy GEMINI_API_KEY environment variable (GOOGLE_API_KEY takes precedence).
Explicit credentials use credential-based authentication instead.
Explicit project/location use Application Default Credentials.
GOOGLE_APPLICATION_CREDENTIALS takes precedence over an API key from the environment.
credentials : Credentials | None Default: None
The credentials to use for authentication when calling the Google Cloud APIs. Credentials can
be obtained from environment variables and default credentials. For more information, see
Set up Application Default Credentials.
Credentials that require scopes are automatically scoped with
https://www.googleapis.com/auth/cloud-platform.
The Google Cloud project ID to use for quota. Can be obtained from environment variables
(for example, GOOGLE_CLOUD_PROJECT).
The location to send API requests to, for example us-central1 (a single region) or
global. 'us' and 'eu' are multi-region values routed to the aiplatform.{us,eu}.rep.googleapis.com
data-residency endpoints. Model availability differs between single regions, multi-regions, and
global — see the
Vertex AI locations docs.
Can be obtained from the GOOGLE_CLOUD_LOCATION environment variable.
client : Client | None Default: None
A pre-initialized client to use. Stored as-is without forcing vertexai=True, so a
client built for the Gemini Developer API is used on that transport even though name
stays 'google-cloud'. GoogleModel reads the transport off the client rather than the
provider name, so it routes correctly either way.
http_client : AsyncHTTPClient | None Default: None
An existing httpx2.AsyncClient or legacy httpx.AsyncClient to use for making HTTP requests.
The base URL for the Google Cloud API.
retry_options : HttpRetryOptions | None Default: None
HTTP retry options for transient errors (429, 5xx, etc.).
See google.genai.types.HttpRetryOptions for available fields.
Bases: _OpenAICompatibleProvider
Provider for OpenAI API.
def __init__(*, openai_client: AsyncOpenAI) -> None
def __init__(
base_url: str | None = None,
api_key: str | None = None,
openai_client: None = None,
http_client: _OpenAIHTTPClient | None = None,
) -> None
Create a new OpenAI provider.
The base url for the OpenAI requests. If not provided, the OPENAI_BASE_URL environment variable
will be used if available. Otherwise, defaults to OpenAI’s base url.
The API key to use for authentication, if not provided, the OPENAI_API_KEY environment variable
will be used if available.
openai_client : AsyncOpenAI | None Default: None
An existing
AsyncOpenAI
client to use. If provided, base_url, api_key, and http_client must be None.
http_client : _OpenAIHTTPClient | None Default: None
An existing httpx2.AsyncClient or legacy httpx.AsyncClient to use for making HTTP requests.
OpenAI Codex subscription-auth provider: OAuth flow primitives, credential refresh, wire dialect.
Core owns the protocol primitives (PKCE context, authorization URL, code exchange, refresh) and,
because the pinned redirect URI makes every login a localhost callback, the one-shot callback
catcher (exchange_code_from_callback()); browser opening and persistent credential storage
belong to applications and harnesses.
The authorization-code + PKCE redirect flow is the only login flow the public Codex client
supports: its registration pins the redirect URI to http://localhost:1455/auth/callback
(exact-match, probed live 2026-08-25), and the auth service serves no device-authorization
endpoint. exchange_code_from_callback() serves this exact redirect URI and exchanges the
authorization code.
Bases: _CredentialsError
Refreshing Codex credentials against the token endpoint failed.
When the underlying error is invalid_grant, the stored grant is no longer usable and a fresh
authorization is required (locally: rerun codex login; in an app: rerun your connect flow).
The error message.
Type: str Default: message
The name of the model associated with the error.
Type: str Default: model_name
Bases: _CredentialsError, OpenAIError
Rotated credentials were updated in memory but the persistence callback raised.
The in-memory credentials are current and were handed to the callback before it failed; the error surfaces so callers do not mistake a failed save for durability.
The error message.
Type: str Default: message
The name of the model associated with the error.
Type: str Default: model_name
Codex subscription credentials.
The tokens are excluded from repr so a logged or traceback-embedded instance does not leak them.
@classmethod
def from_codex_cli_auth(cls, data: Mapping[str, Any]) -> Self
Parse the Codex CLI ~/.codex/auth.json shape ({'tokens': {...}}).
Bases: Protocol
Application-owned storage for the credentials, so refreshed tokens outlive the process.
The provider owns the credential lifecycle (expiry checks, refresh, single-flight) and calls
this only to read and write the stored set: load() on first use, and save() after tokens
are refreshed.
Refresh tokens are single-use, so before refreshing, the provider re-reads storage with
load() and adopts newer credentials if another process using the same store already
refreshed them. That is best-effort, not mutual exclusion: two processes can still race
between load() and save().
Conformance is structural, but implementations are encouraged to subclass the protocol explicitly so type checkers verify the method signatures.
@async
def load() -> OpenAICodexCredentials
Return the currently stored credentials.
OpenAICodexCredentials
@async
def save(credentials: OpenAICodexCredentials) -> None
Durably replace the stored credentials with a freshly rotated set.
Bases: OAuthFlow[OpenAICodexCredentials]
Pure authorization-code + PKCE context for the OpenAI Codex public client.
This is the only login flow the public client supports (no device flow; redirect URI pinned to
localhost:1455, probed exact-match). Construction does no I/O: build the context anywhere,
send the user to authorization_url(), then let exchange_code_from_callback() receive the
redirect on localhost and exchange its code. The browser and credential storage stay
caller-owned.
The S256 PKCE challenge derived from code_verifier.
Type: str
@async
def exchange_code_from_callback() -> CredentialsT
Serve redirect_uri for one authorization callback, then exchange the received code.
Binds the host and port from redirect_uri with a one-shot local HTTP server, ignores
requests that don’t carry this flow’s state, and raises
UserError when the provider reports an authorization
error instead of a code (e.g. the user clicked Deny). Callers wanting a time limit can wrap
the call in anyio.fail_after.
CredentialsT
def __init__(*, redirect_uri: str = _REDIRECT_URI, state: str | None = None) -> None
Create a new flow context. Construction does no I/O.
redirect_uri : str Default: _REDIRECT_URI
Where the authorization code is delivered. The public client’s
registration pins this to http://localhost:1455/auth/callback (exact-match),
so leave the default unchanged when using the public client.
The CSRF token bound to the callback; auto-generated when None.
def authorization_url(
*,
scope: str | None = None,
extra_params: Mapping[str, str] | None = None,
) -> str
The URL to send the user to. Note the public client pins redirects to localhost.
The OAuth scopes to request; None means the standard Codex login scopes.
Additional query parameters, merged over the defaults (so they can also
override them), except client_id and redirect_uri: exchange_code() always
posts the public client id and the flow’s redirect_uri, so overriding either
here would make the authorization code unusable. The production Codex login’s
id_token_add_organizations=true and codex_cli_simplified_flow=true are sent by
default: without the former, the id_token can omit the account id for multi-org
accounts (live-verified 2026-08-25).
@async
def exchange_code(code: str) -> OpenAICodexCredentials
Exchange an authorization code for credentials (call this in your callback handler).
OpenAICodexCredentials
Bases: _OpenAICompatibleProvider
Provider for OpenAI Codex subscription authentication.
Wraps the standard OpenAIProvider machinery pointed at the Codex backend, injecting Codex
OAuth credentials instead of API keys. One provider instance carries one set of credentials
(there is no process-global cache). The instance binds its refresh lock to the first event
loop that awaits a request, so do not reuse it across loops.
provider = OpenAICodexProvider(credential_source=YourCredentialStore())
agent = Agent('openai-codex:gpt-5.6-luna', provider=provider)
The namespace used to fully qualify model IDs routed through this provider.
Type: str
The credentials currently held in memory, rotated in place by refreshes.
Type: OpenAICodexCredentials
@staticmethod
def realtime_model_profile(model_name: str) -> RealtimeModelProfile | None
The realtime model profile for the named model, if available.
RealtimeModelProfile | None
def __init__(
credentials: OpenAICodexCredentials | None = None,
*,
credential_source: OpenAICodexCredentialSource | None = None,
openai_client: AsyncOpenAI | None = None,
http_client: _OpenAIHTTPClient | None = None,
) -> None
Create a new OpenAI Codex provider.
credentials : OpenAICodexCredentials | None Default: None
The subscription credentials to inject. If both this and
credential_source are omitted, they are loaded read-only from the Codex
CLI’s auth.json (honors CODEX_HOME), which never writes the file: refreshed
tokens then live in memory only. Pydantic AI never falls back to OPENAI_API_KEY.
credential_source : OpenAICodexCredentialSource | None Default: None
Application-owned storage for the credentials, so refreshed tokens
are persisted between runs; see
OpenAICodexCredentialSource.
Mutually exclusive with credentials.
openai_client : AsyncOpenAI | None Default: None
An existing AsyncOpenAI client to use as-is. Opts out of credential
injection entirely; credentials, credential_source, and http_client must
be None.
http_client : _OpenAIHTTPClient | None Default: None
An existing httpx2.AsyncClient to use. Must be dedicated to this
provider (no auth of its own): the provider attaches its credential-injecting auth to it, and
sharing a client between providers would mix their credentials. The auth only
injects credentials on HTTPS requests to the Codex host, so the client can safely
be reused for other destinations.
Bases: Provider[AsyncClient]
Provider for xAI API (native xAI SDK).
The resolved API key, or None when the provider was built from a pre-configured xai_client.
The gRPC AsyncClient doesn’t expose its key, so this returns the one resolved from the api_key
argument or XAI_API_KEY. Used by transports that authenticate outside the SDK, e.g.
XaiRealtimeModel’s WebSocket Authorization header.
The custom api_host this provider was configured with, or None.
Read by XaiRealtimeModel to reject a custom host
it can’t yet honor: the realtime WebSocket derives its URL from base_url, not the gRPC channel
target that api_host sets.
def __init__(
*,
api_key: str | None = None,
api_host: str | None = None,
timeout: float | None = None,
metadata: tuple[tuple[str, str], ...] | None = None,
) -> None
def __init__(*, xai_client: AsyncClient) -> None
Create a new xAI provider.
The API key to use for authentication, if not provided, the XAI_API_KEY environment variable
will be used if available.
The API host to use for the xAI SDK client.
The client-level default timeout for the xAI SDK client, in seconds, applied to all requests
made through it. The xAI SDK does not support per-request timeouts, so ModelSettings.timeout is
not supported and has no effect.
gRPC metadata to attach to every request the xAI SDK client makes, forwarded to
xai_sdk.AsyncClient. This is client-scoped, not per-request, so it applies
to every request made through the provider. The canonical use is xAI prompt-cache sticky routing via
metadata=(('x-grok-conv-id', '<conversation-id>'),); see the
xAI prompt-caching docs.
Because it is client-scoped, a provider configured with conversation-specific metadata (e.g. a fixed
x-grok-conv-id) must not be shared between unrelated conversations. Ignored when xai_client is
passed.
xai_client : AsyncClient | None Default: None
An existing xai_sdk.AsyncClient to use. This takes precedence over api_key, api_host,
timeout, and metadata.
Bases: _OpenAICompatibleProvider
Provider for DeepSeek API.
Bases: JsonSchemaTransformer
Transforms schemas to the subset supported by Bedrock structured outputs.
The transformer is applied to Bedrock tool and output schemas during request customization. Strict-mode rewrites are applied when:
NativeOutputis used as theoutput_typeof the Agent.BedrockConverseModelforces native output schemas tostrict=Truebefore request customization.strict=Trueis set explicitly on a Tool.
Like AnthropicJsonSchemaTransformer, Bedrock does not infer strict tool mode
from strict=None. Strict tool definitions are opt-in: callers must set
strict=True explicitly. This avoids silently changing large toolsets into
strict toolsets, which can exceed Anthropic/Bedrock’s 20 strict-tools-per-request
limit, and avoids applying potentially lossy strict-mode schema rewrites unless
requested.
When strict=True, additionalProperties: false is injected on objects and keys
Bedrock rejects are removed from the schema and re-emitted into the field’s
description so the model still has the hint.
Bases: ModelProfile
Profile for models used with BedrockModel.
ALL FIELDS MUST BE bedrock_ PREFIXED SO YOU CAN MERGE THEM WITH OTHER MODELS.
Default: False.
Type: bool
Default: 'text'.
Type: Literal[‘text’, ‘json’]
Default: False.
Type: bool
Default: False.
Type: bool
Default: False.
Type: bool
Default: frozenset({'image'}).
Type: frozenset[Literal[‘image’, ‘document’, ‘video’]]
Content-block kinds that this model accepts in the same user message as a toolResult block.
pydantic-ai merges consecutive user turns into one Bedrock message, which can place a toolResult
alongside a following turn’s text/attachment. Some models reject that: Anthropic rejects documents
and video next to a toolResult, while Llama and Mistral reject any content sharing the turn (the
toolResult must be alone). When a merge would co-locate a toolResult with a kind not listed here,
the adapter splits the turns and separates them with a synthetic assistant message (Bedrock re-merges
consecutive same-role turns, so a bare split doesn’t suffice). See https://github.com/pydantic/pydantic-ai/issues/6081.
Default: all kinds (no restriction); the model receives merged turns unchanged.
Type: frozenset[Literal[‘text’, ‘image’, ‘document’, ‘video’]]
Whether this model accepts a conversation that starts with an assistant message.
Bedrock’s Converse API requires that a conversation start with a user message for most model
families (Amazon Nova, Meta Llama, Mistral, Cohere, AI21, Writer, …), which reject a leading
assistant turn with "A conversation must start with a user message...". Anthropic and Qwen
models accept a leading assistant turn, so for them we don’t need to synthesize a placeholder
user message when message_history starts with a ModelResponse.
Verified against Bedrock us-east-1 on 2026-07-03.
Default: False (strict — synthesize a leading user message when history starts with an
assistant turn).
Type: bool
Whether this model accepts the status field on a toolResult block in Bedrock’s Converse API.
Most families accept (and pydantic-ai emits) status: 'success'/'error' on toolResult blocks, but
Writer Palmyra rejects it ("This model doesn't support the status field. Remove status and try again."),
so the field is omitted for it. Verified against Bedrock us-east-1.
Default: True.
Type: bool
Whether this model accepts strict: true on toolSpec in Bedrock’s Converse API.
Tracked separately from supports_json_schema_output (which gates NativeOutput /
outputConfig) because AWS could in principle ship a model that supports one without the
other; today both features track the same per-model allowlist per the Bedrock structured-output
docs: https://docs.aws.amazon.com/bedrock/latest/userguide/structured-output.html.
Default: False.
Type: bool
Which thinking API shape to use for unified thinking translation.
'anthropic': Uses{'thinking': {'type': 'adaptive'}}for 4.6+ models, or{'thinking': {'type': 'enabled', 'budget_tokens': N}}for older models.'openai': Uses{'reasoning_effort': 'low'|'medium'|'high'}'qwen': Uses{'reasoning_config': 'low'|'high'}None: No unified thinking support.
Default: None.
Type: Literal[‘anthropic’, ‘openai’, ‘qwen’] | None
Whether this model accepts {'thinking': {'type': 'adaptive'}} (Sonnet 4.6+, Opus 4.6+).
Only meaningful for the 'anthropic' variant. When False, the variant falls back to
{'type': 'enabled', 'budget_tokens': N} for pre-4.6 models.
Default: False.
Type: bool
Whether this model emits output_config.effort on Bedrock Converse (Sonnet 4.6+, Opus 4.6+).
Only meaningful for the 'anthropic' variant AND only honored alongside
bedrock_supports_adaptive_thinking=True. Bedrock has not been verified to accept
output_config.effort on the legacy {'type': 'enabled', 'budget_tokens': N} path
(e.g. Opus 4.5), so the translator skips it there even though the direct Anthropic
API accepts it. Effort lives at additionalModelRequestFields.output_config.effort
(a sibling of thinking, not inside it).
Default: False.
Type: bool
How the unified top_k setting is placed in additionalModelRequestFields.
Bedrock’s Converse inferenceConfig has no topK field, so top_k must travel in the
model-specific additionalModelRequestFields blob, where the shape differs per family
(and Bedrock 400s on an unrecognized key rather than ignoring it):
'anthropic': flat{'top_k': N}'nova': nested{'inferenceConfig': {'topK': N}}None:top_kis silently dropped (Llama/Mistral/DeepSeek/Jamba don’t accept it on Converse; Cohere’skand Qwen’s key are unverified on Converse, so they stay here too).
Type: Literal[‘anthropic’, ‘nova’] | None
Whether this model is served by the Bedrock Converse API. Default: True.
Set to False for models that Bedrock serves only through the Mantle OpenAI-compatible API (today,
the proprietary OpenAI GPT models); BedrockConverseModel raises at construction so the user gets an
actionable pointer to BedrockMantleProvider instead of an opaque Converse error at request time.
Type: bool
Bases: Provider[BaseClient]
Provider for AWS Bedrock.
The boto3 client used to make requests to the Bedrock API.
Type: BaseClient
def __init__(*, bedrock_client: BaseClient) -> None
def __init__(
*,
api_key: str,
base_url: str | None = None,
region_name: str | None = None,
profile_name: str | None = None,
aws_read_timeout: float | None = None,
aws_connect_timeout: float | None = None,
) -> None
def __init__(
*,
aws_access_key_id: str | None = None,
aws_secret_access_key: str | None = None,
aws_session_token: str | None = None,
base_url: str | None = None,
region_name: str | None = None,
profile_name: str | None = None,
aws_read_timeout: float | None = None,
aws_connect_timeout: float | None = None,
) -> None
Initialize the Bedrock provider.
bedrock_client : BaseClient | None Default: None
A boto3 client for Bedrock Runtime. If provided, other arguments are ignored.
The AWS access key ID. If not set, the AWS_ACCESS_KEY_ID environment variable will be used if available.
The AWS secret access key. If not set, the AWS_SECRET_ACCESS_KEY environment variable will be used if available.
The AWS session token. If not set, the AWS_SESSION_TOKEN environment variable will be used if available.
The API key for Bedrock client. Can be used instead of aws_access_key_id, aws_secret_access_key, and aws_session_token. If not set, the AWS_BEARER_TOKEN_BEDROCK environment variable will be used if available.
The base URL for the Bedrock client.
The AWS region name. If not set, the AWS_DEFAULT_REGION environment variable will be used if available.
The AWS profile name.
The read timeout for Bedrock client.
The connect timeout for Bedrock client.
def bedrock_anthropic_model_profile(model_name: str) -> ModelProfile | None
Get the model profile for an Anthropic model used via Bedrock.
def bedrock_amazon_model_profile(model_name: str) -> ModelProfile | None
Get the model profile for an Amazon model used via Bedrock.
def bedrock_deepseek_model_profile(model_name: str) -> ModelProfile | None
Get the model profile for a DeepSeek model used via Bedrock.
def bedrock_meta_model_profile(model_name: str) -> ModelProfile | None
Get the model profile for a Meta Llama model used via Bedrock.
def bedrock_mistral_model_profile(model_name: str) -> ModelProfile | None
Get the model profile for a Mistral model used via Bedrock.
def bedrock_qwen_model_profile(model_name: str) -> ModelProfile | None
Get the model profile for a Qwen model used via Bedrock.
def bedrock_google_model_profile(model_name: str) -> ModelProfile | None
Get the model profile for a Google model used via Bedrock.
def bedrock_zai_model_profile(model_name: str) -> ModelProfile | None
Get the model profile for a Z.AI (Zhipu) GLM model used via Bedrock.
def bedrock_moonshotai_model_profile(model_name: str) -> ModelProfile | None
Get the model profile for a Moonshot AI Kimi model used via Bedrock.
Registered for both the moonshot. and moonshotai. Bedrock provider prefixes.
def bedrock_writer_model_profile(model_name: str) -> ModelProfile | None
Get the model profile for a Writer Palmyra model used via Bedrock.
def bedrock_minimax_model_profile(model_name: str) -> ModelProfile | None
Get the model profile for a MiniMax model used via Bedrock.
def bedrock_nvidia_model_profile(model_name: str) -> ModelProfile | None
Get the model profile for an NVIDIA model used via Bedrock.
def bedrock_openai_model_profile(model_name: str) -> ModelProfile | None
Get the model profile for an OpenAI model used via Bedrock Converse.
Bases: OpenAIModelProfile
Profile for OpenAI models served through Amazon Bedrock Mantle.
Which Mantle endpoint family serves this model, selecting the model class and base URL.
Type: BedrockMantleInterface
Bases: _OpenAICompatibleProvider
Provider for the Amazon Bedrock Mantle OpenAI-compatible API.
def __init__(*, openai_client: AsyncOpenAI) -> None
def __init__(
*,
region_name: str | None = None,
base_url: str | None = None,
api_key: str | None = None,
aws_access_key_id: str | None = None,
aws_secret_access_key: str | None = None,
aws_session_token: str | None = None,
profile_name: str | None = None,
http_client: _OpenAIHTTPClient | None = None,
) -> None
Create a Bedrock Mantle provider.
The AWS region used to construct the default bedrock-mantle.{region}.api.aws
origin. If not set, the AWS_DEFAULT_REGION or AWS_REGION environment variable is used.
A Mantle base URL. Its origin (scheme + host, with any /openai/v1 or /v1 suffix
stripped) is used to route between /v1 and /openai/v1 per model, the same as region_name.
A Bedrock API key. If omitted, AWS_BEARER_TOKEN_BEDROCK is used. Use this or the
aws_* credentials, not both.
The AWS access key ID for SigV4 authentication.
The AWS secret access key for SigV4 authentication.
The AWS session token for SigV4 authentication.
The AWS profile name for SigV4 authentication.
openai_client : AsyncOpenAI | None Default: None
An existing OpenAI client. If provided, no other argument may be set; its base
URL’s origin is used to derive both the /v1 and /openai/v1 endpoints (preserving its
auth and transport) so every interface routes correctly.
http_client : _OpenAIHTTPClient | None Default: None
An existing httpx2.AsyncClient or legacy httpx.AsyncClient used to make requests.
def bedrock_mantle_model_profile(model_name: str) -> ModelProfile
Resolve the profile for an OpenAI model served through Bedrock Mantle.
The OpenAI-compatible endpoint family a Bedrock Mantle model is served on.
'chat': Chat Completions at/v1/chat/completions(GPT-OSS Safeguard).'responses': Responses at/v1/responses(GPT-OSS).'openai-responses': Responses at/openai/v1/responses, the OpenAI-model-specific path (GPT-5.4+).
Default: Literal['chat', 'responses', 'openai-responses']
Bases: Provider[AsyncGroq]
Provider for Groq API.
def __init__(*, groq_client: AsyncGroq | None = None) -> None
def __init__(
*,
api_key: str | None = None,
base_url: str | None = None,
http_client: httpx.AsyncClient | None = None,
) -> None
Create a new Groq provider.
The API key to use for authentication, if not provided, the GROQ_API_KEY environment variable
will be used if available.
The base url for the Groq requests. If not provided, the GROQ_BASE_URL environment variable
will be used if available. Otherwise, defaults to Groq’s base url.
groq_client : AsyncGroq | None Default: None
An existing
AsyncGroq
client to use. If provided, api_key and http_client must be None.
http_client : httpx.AsyncClient | None Default: None
An existing AsyncClient to use for making HTTP requests.
def groq_moonshotai_model_profile(model_name: str) -> ModelProfile | None
Get the model profile for an MoonshotAI model used with the Groq provider.
def meta_groq_model_profile(model_name: str) -> ModelProfile | None
Get the model profile for a Meta model used with the Groq provider.
Bases: _OpenAICompatibleProvider
Provider for Azure OpenAI API.
See https://azure.microsoft.com/en-us/products/ai-foundry for more information.
The Azure resource endpoint used to derive service-specific URLs.
Type: str
The Azure resource key, for transports that authenticate with one.
Raises UserError when the provider has no key, i.e. it
was built from a Microsoft Entra ID client (azure_ad_token / azure_ad_token_provider).
Type: str
The Azure AI Voice Live endpoint (AZURE_VOICELIVE_ENDPOINT, else the Azure OpenAI endpoint).
Type: str
The Azure AI Voice Live key (AZURE_VOICELIVE_API_KEY, else the Azure OpenAI key).
Type: str
The Azure AI Voice Live API version (AZURE_VOICELIVE_API_VERSION, else a supported default).
Type: str
@classmethod
def for_realtime(
cls,
*,
azure_endpoint: str | None = None,
api_version: str | None = None,
api_key: str | None = None,
entra_authenticated: bool = False,
http_client: _OpenAIHTTPClient | None = None,
) -> Self
Create an Azure provider for the GA realtime API.
The realtime transport always uses Azure’s /openai/v1 protocol and does not send an
api_version. When neither api_version nor OPENAI_API_VERSION is set, a bare resource
endpoint is therefore normalized to its /openai/v1 form before constructing the provider.
Explicit arguments otherwise follow the same environment fallbacks and validation as the
standard constructor.
The Azure resource endpoint. Falls back to AZURE_OPENAI_ENDPOINT.
The API version for endpoints that require one. Falls back to
OPENAI_API_VERSION.
The Azure resource key. Falls back to AZURE_OPENAI_API_KEY.
entra_authenticated : bool Default: False
Set when every request is authenticated with a Microsoft Entra ID
credential instead of the resource key (see
AzureRealtimeModel(credential=...)).
The key is then neither required nor sent, and api_key raises its usual explanatory
error if anything asks — the same state a provider built from an Entra-authenticated
openai_client lands in.
http_client : _OpenAIHTTPClient | None Default: None
An existing httpx2.AsyncClient or legacy httpx.AsyncClient used to construct the provider client.
def __init__(*, openai_client: AsyncAzureOpenAI) -> None
def __init__(
*,
azure_endpoint: str | None = None,
api_version: str | None = None,
api_key: str | None = None,
voice_live_endpoint: str | None = None,
voice_live_api_key: str | None = None,
voice_live_api_version: str | None = None,
http_client: _OpenAIHTTPClient | None = None,
) -> None
Create a new Azure provider.
The Azure endpoint to use for authentication, if not provided, the AZURE_OPENAI_ENDPOINT
environment variable will be used if available.
The API version to use for authentication, if not provided, the OPENAI_API_VERSION
environment variable will be used if available. Not required (and not sent) when
azure_endpoint targets the Azure OpenAI v1 GA API
(i.e. a path ending in /v1, such as https://<resource>.openai.azure.com/openai/v1/)
or an Azure AI Foundry serverless model endpoint (*.models.ai.azure.com), both of
which reject the api-version query parameter.
The API key to use for authentication, if not provided, the AZURE_OPENAI_API_KEY environment variable
will be used if available.
The Azure AI Voice Live
endpoint, used only by AzureRealtimeModel
with azure_voice_live=True. Voice Live is a distinct Azure resource, so when this is
not provided the AZURE_VOICELIVE_ENDPOINT environment variable is used, and finally
azure_endpoint as a fallback.
The Voice Live API key; falls back to AZURE_VOICELIVE_API_KEY, then api_key.
The Voice Live API version; falls back to AZURE_VOICELIVE_API_VERSION,
then a supported default. Deliberately not derived from api_version, which versions
the Azure OpenAI data plane on an unrelated schedule.
openai_client : AsyncAzureOpenAI | None Default: None
An existing
AsyncAzureOpenAI
client to use. If provided, base_url, api_key, and http_client must be None.
http_client : _OpenAIHTTPClient | None Default: None
An existing httpx2.AsyncClient or legacy httpx.AsyncClient to use for making HTTP requests.
Bases: Provider[AsyncClientV2]
Provider for Cohere API.
def __init__(
*,
api_key: str | None = None,
cohere_client: AsyncClientV2 | None = None,
http_client: httpx.AsyncClient | None = None,
) -> None
Create a new Cohere provider.
The API key to use for authentication, if not provided, the CO_API_KEY environment variable
will be used if available.
cohere_client : AsyncClientV2 | None Default: None
An existing
AsyncClientV2
client to use. If provided, api_key and http_client must be None.
http_client : httpx.AsyncClient | None Default: None
An existing httpx.AsyncClient to use for making HTTP requests.
Bases: Provider[AsyncClient]
Provider for VoyageAI API.
def __init__(*, voyageai_client: AsyncClient) -> None
def __init__(*, api_key: str | None = None) -> None
Create a new VoyageAI provider.
The API key to use for authentication, if not provided, the VOYAGE_API_KEY environment variable
will be used if available.
voyageai_client : AsyncClient | None Default: None
An existing
AsyncClient
client to use. If provided, api_key must be None.
Bases: Provider[Any]
Provider for Sentence Transformers API.
The provider name.
Type: str
The base URL for the provider API.
Type: str
The client for the provider.
Type: Any
Bases: _OpenAICompatibleProvider
Provider for Cerebras API.
def __init__() -> None
def __init__(*, api_key: str) -> None
def __init__(*, api_key: str, http_client: _OpenAIHTTPClient) -> None
def __init__(*, http_client: _OpenAIHTTPClient) -> None
def __init__(*, openai_client: AsyncOpenAI | None = None) -> None
Create a new Cerebras provider.
The API key to use for authentication, if not provided, the CEREBRAS_API_KEY environment variable
will be used if available.
openai_client : AsyncOpenAI | None Default: None
An existing AsyncOpenAI client to use. If provided, api_key and http_client must be None.
http_client : _OpenAIHTTPClient | None Default: None
An existing httpx2.AsyncClient or legacy httpx.AsyncClient to use for making HTTP requests.
Bases: Provider[Mistral]
Provider for Mistral API.
def __init__(*, mistral_client: Mistral | None = None) -> None
def __init__(
*,
api_key: str | None = None,
http_client: AsyncHTTPClient | None = None,
) -> None
Create a new Mistral provider.
The API key to use for authentication, if not provided, the MISTRAL_API_KEY environment variable
will be used if available.
mistral_client : Mistral | None Default: None
An existing Mistral client to use, if provided, api_key and http_client must be None.
The base url for the Mistral requests.
http_client : AsyncHTTPClient | None Default: None
An existing httpx2.AsyncClient or legacy httpx.AsyncClient to use for making HTTP requests.
Bases: _OpenAICompatibleProvider
Provider for Fireworks AI API.
Bases: _OpenAICompatibleProvider
Provider for Together AI API.
Bases: _OpenAICompatibleProvider
Provider for Heroku API.
Bases: Provider[AsyncOpenAI]
Provider for the retired GitHub Models API.
GitHub Models was retired on 2026-07-30; the inference API is no longer available, so this provider is deprecated and will be removed in v3.
def __init__() -> None
def __init__(*, api_key: str) -> None
def __init__(*, api_key: str, http_client: httpx.AsyncClient) -> None
def __init__(*, openai_client: AsyncOpenAI | None = None) -> None
Create a new GitHub Models provider.
The GitHub token to use for authentication. If not provided, the GITHUB_API_KEY
environment variable will be used if available.
openai_client : AsyncOpenAI | None Default: None
An existing AsyncOpenAI client to use. If provided, api_key and http_client must be None.
http_client : httpx.AsyncClient | None Default: None
An existing httpx.AsyncClient to use for making HTTP requests.
Bases: _OpenAICompatibleProvider
Provider for GitHub Copilot.
Routes requests through Copilot’s OpenAI-compatible Chat Completions API at
https://api.githubcopilot.com/chat/completions. Copilot serves Anthropic, OpenAI, Google, xAI and
MoonshotAI models under a subscription, but only the ids whose catalog entry lists /chat/completions
under supported_endpoints are reachable here; xAI’s Grok ids, for one, are served on the Responses
API alone. Which ids you can reach also depends on your plan; list yours with
GET https://api.githubcopilot.com/models.
This is not GitHubProvider, which served the
retired GitHub Models API.
def __init__(*, openai_client: AsyncOpenAI) -> None
def __init__(
*,
api_key: str | None = None,
base_url: str | None = None,
openai_client: None = None,
http_client: _OpenAIHTTPClient | None = None,
) -> None
Create a new GitHub Copilot provider.
The Copilot token to authenticate with. Defaults to the GITHUB_COPILOT_API_KEY
environment variable, then to GITHUB_COPILOT_API_TOKEN and COPILOT_GITHUB_TOKEN,
the names GitHub’s own tooling uses. The general-purpose GITHUB_TOKEN, GH_TOKEN
and GITHUB_API_KEY variables are deliberately not read, so a token meant for the
GitHub API is never sent to Copilot.
The base URL of the Copilot inference API, e.g. for an enterprise host or a
local proxy. Defaults to the GITHUB_COPILOT_BASE_URL, COPILOT_API_URL or
GITHUB_COPILOT_API_BASE environment variable, then to https://api.githubcopilot.com.
openai_client : AsyncOpenAI | None Default: None
An existing AsyncOpenAI client to use. Its base_url must already point
at the Copilot inference API, and it is used as-is, without the Copilot client
headers. If provided, api_key, base_url and http_client must be None.
http_client : _OpenAIHTTPClient | None Default: None
An existing httpx2.AsyncClient or legacy httpx.AsyncClient to use for making HTTP requests.
Bases: _OpenAICompatibleProvider
Provider for OpenRouter API.
def __init__(*, openai_client: AsyncOpenAI) -> None
def __init__(
*,
api_key: str | None = None,
app_url: str | None = None,
app_title: str | None = None,
openai_client: None = None,
http_client: _OpenAIHTTPClient | None = None,
) -> None
Configure the provider with either an API key or prebuilt client.
OpenRouter API key. Falls back to OPENROUTER_API_KEY
when omitted and required unless openai_client is provided.
Optional url for app attribution. Falls back to
OPENROUTER_APP_URL when omitted.
Optional title for app attribution. Falls back to
OPENROUTER_APP_TITLE when omitted.
openai_client : AsyncOpenAI | None Default: None
Existing AsyncOpenAI client to reuse instead of
creating one internally.
http_client : _OpenAIHTTPClient | None Default: None
Custom httpx2.AsyncClient or legacy httpx.AsyncClient to pass into the
AsyncOpenAI constructor when building a client.
UserError— If no API key is available and noopenai_clientis provided.
Bases: _OpenAICompatibleProvider
Provider for Vercel AI Gateway API.
Bases: Provider[AsyncInferenceClient]
Provider for Hugging Face.
def __init__(*, base_url: str, api_key: str | None = None) -> None
def __init__(*, provider_name: str, api_key: str | None = None) -> None
def __init__(*, hf_client: AsyncInferenceClient, api_key: str | None = None) -> None
def __init__(
*,
hf_client: AsyncInferenceClient,
base_url: str,
api_key: str | None = None,
) -> None
def __init__(
*,
hf_client: AsyncInferenceClient,
provider_name: str,
api_key: str | None = None,
) -> None
def __init__(*, api_key: str | None = None) -> None
Create a new Hugging Face provider.
The base url for the Hugging Face requests.
The API key to use for authentication, if not provided, the HF_TOKEN environment variable
will be used if available.
hf_client : AsyncInferenceClient | None Default: None
An existing
AsyncInferenceClient
client to use. If not provided, a new instance will be created.
http_client : AsyncClient | None Default: None
(currently ignored) An existing httpx.AsyncClient to use for making HTTP requests.
Name of the provider to use for inference. available providers can be found in the HF Inference Providers documentation.
defaults to “auto”, which will select the first available provider for the model, the first of the providers available for the model, sorted by the user’s order in https://hf.co/settings/inference-providers.
If base_url is passed, then provider_name is not used.
Bases: _OpenAICompatibleProvider
Provider for MoonshotAI platform (Kimi models).
Bases: _OpenAICompatibleProvider
Provider for local or remote Ollama API.
def __init__(
base_url: str | None = None,
api_key: str | None = None,
openai_client: AsyncOpenAI | None = None,
http_client: _OpenAIHTTPClient | None = None,
) -> None
Create a new Ollama provider.
The base url for the Ollama requests. If not provided, the OLLAMA_BASE_URL environment variable
will be used if available.
The API key to use for authentication, if not provided, the OLLAMA_API_KEY environment variable
will be used if available.
openai_client : AsyncOpenAI | None Default: None
An existing
AsyncOpenAI
client to use. If provided, base_url, api_key, and http_client must be None.
http_client : _OpenAIHTTPClient | None Default: None
An existing httpx2.AsyncClient or legacy httpx.AsyncClient to use for making HTTP requests.
Bases: _OpenAICompatibleProvider
Provider for LiteLLM API.
def __init__(*, api_key: str | None = None, api_base: str | None = None) -> None
def __init__(
*,
api_key: str | None = None,
api_base: str | None = None,
http_client: _OpenAIHTTPClient,
) -> None
def __init__(*, openai_client: AsyncOpenAI) -> None
Initialize a LiteLLM provider.
API key for the model provider. If None, LiteLLM will try to get it from environment variables.
Base URL for the model provider. Use this for custom endpoints or self-hosted models.
openai_client : AsyncOpenAI | None Default: None
Pre-configured OpenAI client. If provided, other parameters are ignored.
http_client : _OpenAIHTTPClient | None Default: None
Custom HTTP client to use.
Bases: _OpenAICompatibleProvider
Provider for local or remote vLLM API.
def __init__(*, openai_client: AsyncOpenAI) -> None
def __init__(
*,
base_url: str | None = None,
api_key: str | None = None,
openai_client: None = None,
http_client: _OpenAIHTTPClient | None = None,
) -> None
Create a new vLLM provider.
The base url for the vLLM requests. If not provided, the VLLM_BASE_URL environment variable
will be used if available.
The API key to use for authentication, if not provided, the VLLM_API_KEY environment variable
will be used if available.
openai_client : AsyncOpenAI | None Default: None
An existing
AsyncOpenAI
client to use. If provided, base_url, api_key, and http_client must be None.
http_client : _OpenAIHTTPClient | None Default: None
An existing httpx2.AsyncClient or legacy httpx.AsyncClient to use for making HTTP requests.
Bases: _OpenAICompatibleProvider
Provider for Nebius AI Studio API.
Bases: _OpenAICompatibleProvider
Provider for OVHcloud AI Endpoints.
Bases: _OpenAICompatibleProvider
Provider for Crusoe Serverless Inference API.
def __init__() -> None
def __init__(*, api_key: str) -> None
def __init__(*, api_key: str, http_client: _OpenAIHTTPClient) -> None
def __init__(*, http_client: _OpenAIHTTPClient) -> None
def __init__(*, openai_client: AsyncOpenAI | None = None) -> None
Create a new Crusoe provider.
The API key to use for authentication, if not provided, the CRUSOE_API_KEY environment
variable will be used if available.
openai_client : AsyncOpenAI | None Default: None
An existing AsyncOpenAI client to use. If provided, api_key and http_client must be None.
http_client : _OpenAIHTTPClient | None Default: None
An existing httpx2.AsyncClient or legacy httpx.AsyncClient to use for making HTTP requests.
Bases: _OpenAICompatibleProvider
Provider for Alibaba Cloud Model Studio (DashScope) OpenAI-compatible API.
Bases: _OpenAICompatibleProvider
Provider for SambaNova AI models.
SambaNova uses an OpenAI-compatible API.
Return the provider name.
Type: str
Return the base URL.
Type: str
Return the AsyncOpenAI client.
Type: AsyncOpenAI
@staticmethod
def model_profile(model_name: str) -> ModelProfile | None
Get model profile for SambaNova models.
SambaNova serves models from multiple families including Meta Llama, DeepSeek, Qwen, and Mistral. Model profiles are matched based on model name prefixes.
def __init__(
*,
api_key: str | None = None,
base_url: str | None = None,
openai_client: AsyncOpenAI | None = None,
http_client: _OpenAIHTTPClient | None = None,
) -> None
Initialize SambaNova provider.
SambaNova API key. If not provided, reads from SAMBANOVA_API_KEY env var.
Custom API base URL. Defaults to https://api.sambanova.ai/v1
openai_client : AsyncOpenAI | None Default: None
Optional pre-configured OpenAI client
http_client : _OpenAIHTTPClient | None Default: None
Optional custom httpx2.AsyncClient or legacy httpx.AsyncClient for making HTTP requests.
UserError— If API key is not provided and SAMBANOVA_API_KEY env var is not set
Bases: _OpenAICompatibleProvider
Provider for Snowflake Cortex.
Routes requests through Snowflake’s OpenAI-compatible Chat Completions API at
https://<account>.snowflakecomputing.com/api/v2/cortex/v1/chat/completions, which serves
Claude, GPT, Llama, Mistral, DeepSeek, and Snowflake’s own models. All inference runs inside
the customer’s Snowflake account, so data never leaves the Snowflake security perimeter.
def __init__(*, openai_client: AsyncOpenAI) -> None
def __init__(
*,
account: str | None = None,
token: str | None = None,
base_url: str | None = None,
openai_client: None = None,
http_client: _OpenAIHTTPClient | None = None,
) -> None
Create a new Snowflake provider.
The Snowflake account identifier,
e.g. myorg-myaccount. Defaults to the SNOWFLAKE_ACCOUNT environment variable.
A Snowflake programmatic access token,
OAuth token, or key-pair JWT, sent as Authorization: Bearer <token>.
Defaults to the SNOWFLAKE_TOKEN environment variable.
The base URL of the Cortex REST API, e.g. when connecting through
private connectivity.
Defaults to https://<account>.snowflakecomputing.com/api/v2/cortex/v1.
openai_client : AsyncOpenAI | None Default: None
An existing AsyncOpenAI client to use. Its base_url must already
point at the Cortex REST API. If provided, account, token, base_url, and
http_client must be None.
http_client : _OpenAIHTTPClient | None Default: None
An existing httpx2.AsyncClient or legacy httpx.AsyncClient to use for making HTTP requests.
Bases: _OpenAICompatibleProvider
Provider for Z.AI (Zhipu AI) API.
Z.AI provides GLM models with support for thinking/reasoning mode and preserved thinking across turns.
def __init__() -> None
def __init__(*, api_key: str) -> None
def __init__(*, api_key: str, http_client: _OpenAIHTTPClient) -> None
def __init__(*, http_client: _OpenAIHTTPClient) -> None
def __init__(*, openai_client: AsyncOpenAI | None = None) -> None
Create a new Z.AI provider.
The API key to use for authentication, if not provided, the ZAI_API_KEY environment variable
will be used if available.
openai_client : AsyncOpenAI | None Default: None
An existing AsyncOpenAI client to use. If provided, api_key and http_client must be None.
http_client : _OpenAIHTTPClient | None Default: None
An existing httpx2.AsyncClient or legacy httpx.AsyncClient to use for making HTTP requests.