OpenAI

Pydantic AI connects an agent to two families of OpenAI voice models, each spoken to over its own protocol by its own model class:

• OpenAILiveModel connects an agent to the GPT-Live family, over the GPT-Live API. The Live model runs the spoken conversation and hands the thinking to a backend model that Pydantic AI configures with the agent’s instructions and tools (see How GPT-Live works).
• OpenAIRealtimeModel connects an agent to OpenAI’s gpt-realtime family of native speech-to-speech models, over the Realtime API. Live is a separate protocol, not a model served by the Realtime API.

You rarely need to choose between the classes yourself: an openai: model name picks the right one (see Model names). Start with the realtime quickstart for the shape of a session, or the text-to-audio example for gpt-realtime.

Setup

To use OpenAI realtime models, install pydantic-ai-slim with the openai-realtime optional group, which bundles the openai package together with the realtime WebSocket transport. GPT-Live’s event types arrived in openai 3.12; the group currently requires openai>=3.31.0:

pip

Terminal

pip install "pydantic-ai-slim[openai-realtime]"

uv

Terminal

uv add "pydantic-ai-slim[openai-realtime]"

Set OPENAI_API_KEY as described in the OpenAI model documentation. Authentication and base URL come from provider, mirroring OpenAIChatModel. The default provider='openai' reads the environment; pass an OpenAIProvider for a custom key or base URL. The realtime WebSocket opens separately, so a custom provider httpx client is not used for it. Sessions run over a server-side WebSocket by default; for browser voice, the browser can exchange media directly over WebRTC while your backend runs the agent (see Deployment).

Model names

Family	Model class	Example model names	Protocol
GPT-Live	OpenAILiveModel	gpt-live-1	GPT-Live API (/live/sessions)
gpt-realtime	OpenAIRealtimeModel	gpt-realtime, gpt-realtime-2.1, gpt-realtime-2.1-mini	Realtime API (/realtime)

Model availability and aliases can change; use the official OpenAI model documentation as the canonical model list.

The provider prefix alone cannot tell the two protocols apart, so the model name picks the class: an openai: (or gateway/openai:) model name starting with gpt-live routes to OpenAILiveModel, and every other name routes to OpenAIRealtimeModel:

from pydantic_ai import Agent

agent = Agent('openai:gpt-6.1-sol', instructions='You are a helpful voice assistant.')

# GPT-Live, through `OpenAILiveModel`:
live = agent.realtime('openai:gpt-live-1')

# gpt-realtime, through `OpenAIRealtimeModel`:
realtime = agent.realtime('openai:gpt-realtime')

Two cases don’t route by name:

• The azure: prefix always builds an AzureRealtimeModel, whatever the model name. Azure OpenAI does not serve GPT-Live, and passing an azure provider to OpenAILiveModel raises UserError. See the Azure page.
• Constructing a model class directly skips the name routing: the class you construct is the protocol you get. When you instantiate one for model-level configuration, as the settings examples below do, pick the class for the model’s family.

How GPT-Live works

A Live session is more constrained than a gpt-realtime one. It owns turn-taking entirely, takes an image only for its backend to look at, and bills by the second rather than by the token, so read its feature support before porting code between the two.

The backend model

The backend model, the one that does the delegated work, is a second model name. Pydantic AI picks it from the first of these that names one:

1. openai_live_delegation={'model': ...} in the settings.
2. A model named after a + in the realtime model name: 'openai:gpt-live-1+gpt-6-luna' delegates to gpt-6-luna.
3. The agent’s own model, since the backend runs the agent’s instructions and tools. It counts when it is an OpenAI model reached the same way as the Live model, directly or through the same gateway route, so an agent built on 'openai:gpt-6.1-sol' delegates to gpt-6.1-sol. That holds with defer_model_check=True too: the name is resolved when the session connects, as the agent’s own run would resolve it.
4. 'auto': the model Pydantic AI currently recommends (AUTO_BACKEND_MODEL), which moves as OpenAI releases new models.

There is always a backend, so a session never fails for want of one; pin it (1 or 2) when its behavior needs to stay put.

Live accepts any backend name when the session starts and only tries it when it first delegates. A backend model that doesn’t exist, or that your account can’t use, would then fail every delegation, so the first failure ends the session with a RealtimeError (code live_backend_model_unavailable) naming the model, rather than leaving a call that can talk but never look anything up.

from pydantic_ai import Agent

agent = Agent('openai:gpt-6.1-sol', instructions='You look things up.')

# Delegates to the agent's model, gpt-6.1-sol:
realtime = agent.realtime('openai:gpt-live-1')

# Delegates to gpt-6-luna instead:
realtime = agent.realtime('openai:gpt-live-1+gpt-6-luna')

How delegation works

Live splits one agent across two models:

Piece	Where it goes
The agent’s instructions and system prompt, including those capabilities add	The backend model, which does the work
The agent’s tools	The backend model, advertised as function tools
Web search	The backend model, as its native web_search tool
openai_live_instructions	The Live model, which does the talking

The agent’s instructions never reach the Live model

The prompt of the model the caller hears is openai_live_instructions alone. The agent’s instructions and system prompt, including those passed to agent.realtime() and those capabilities add, go only to the backend. A persona, language, or speaking style written there doesn’t shape what Live says on its own: asked its name with the persona only in the agent’s instructions, the Live model introduced itself as ChatGPT from OpenAI in our testing. Live rewords the backend’s answers in its own voice, so the backend’s style carries into them only in part: rules like “reply in French” or a fixed opening word often didn’t survive into the spoken answer.

Put the work, its rules, and its tools on the agent, and who is speaking and how in openai_live_instructions:

from pydantic_ai import Agent
from pydantic_ai.realtime.openai_live import OpenAILiveModelSettings

agent = Agent(
    # The backend's prompt: the work, its rules, and its tools.
    instructions='You handle order questions for The Terrace. Quote prices exactly as returned.'
)


@agent.tool_plain
async def lookup_order(order_id: str) -> str:
    """Look up an order by its ID."""
    return f'Order {order_id} ships tomorrow.'


realtime = agent.realtime(
    'openai:gpt-live-1',
    model_settings=OpenAILiveModelSettings(
        # The Live model's prompt: who is speaking, and how.
        openai_live_instructions=(
            'You are Sam, the voice of The Terrace. Speak warmly and briefly. '
            'When the caller asks about an order, delegate the lookup and say you are checking.'
        ),
        openai_live_delegation={'model': 'gpt-6.1-sol', 'reasoning_effort': 'low'},
    ),
)

Setting openai_live_instructions replaces the default prompt, which tells the Live model to delegate anything it can’t answer from the conversation, so say when to delegate there too.

When the Live model decides it cannot answer from the conversation alone, it opens a delegation and the backend model takes over, calling the agent’s tools as it goes. Those calls arrive as ordinary ToolCalls, so the session runs them through the same tool loop as every other provider, including validation, retries, dependencies, and capability hooks, and sends the results back. Speech and delegated work run independently, so the Live model can keep talking while that happens rather than leaving dead air.

In history, the round has the shape a standard run gives it: the reply that delegated holds the backend’s tool calls, their results follow it, and then comes the answer Live speaks once the backend has them. What Live says while the backend works (“Okay, checking the forecast … now.”) is part of the reply that delegated, ahead of the results: Live only speaks with them in hand once the backend answers. So that reply, and the results after it, appear in all_messages() once the backend answers, or calls more tools with those results, which are then a response of their own, after them. The backend’s calls land wherever Live is in a sentence, so what Live says in the reply is recorded as one SpeechPart, ahead of the calls, rather than cut in two at each one. Live also finishes what it was saying when the backend answers, before it speaks the answer, and nothing on the wire marks where the answer starts: the reply ends where Live next pauses for half a second, or at anything else that happens first (the user speaking, say). Should Live run straight on into the answer without that pause, the start of the answer is recorded in the reply, ahead of the results.

Live can also delegate to the client rather than to a backend model, leaving your application to answer in prose. Pydantic AI configures backend delegation, so a client delegation is reported as a RealtimeSessionErrorEvent with code live_client_delegation rather than stalling the call silently.

To run the harder reasoning under your own control instead, expose a tool that delegates to a standard agent; that works here exactly as it does on the other providers.

Live has no command to cancel a delegation, so once the backend has the work it runs to completion, tools included, even if the user takes the question back (“never mind”) in the meantime, and the Live model may still speak its result. Put tools that change state behind approval if a retracted request must not go through.

The turn boundary is inferred

Live sends no end-of-response frame and no transcript-done event, so there is nothing on the wire that marks the end of a reply. The connection infers one: once the model has produced no speech for openai_live_turn_silence_ms and no delegated work is outstanding, it reports the turn complete and the session emits RealtimeTurnCompleteEvent. The end of the user’s turn is inferred the same way.

The profile reports synthesizes_turn_boundary=True so an application can tell an inference from a protocol fact (see Provider support for how to read a profile). Treat the turn boundary as a good guess here: a long dramatic pause can end a turn early, and code that must not act on a partial reply should confirm against the transcript. Lower the threshold for snappier turn-taking, raise it when replies contain long silences.

Delegated work suspends the clock. The model goes quiet while the backend thinks, and ending the turn there would finalize a reply that is still coming.

Typed text runs the backend

Live has no client event that puts text into the conversation as the user’s own words, so a string sent with send() goes to the backend as the user’s message, and the backend is run on it: it calls the agent’s tools as it would for a spoken request, and Live speaks its answer. Live is also told what was typed, as silent context, so it knows what it is answering. The text is recorded in history as a UserPromptPart, as on the other providers, and wait_for_reply() returns once Live has spoken the answer:

from pydantic_ai.realtime import RealtimeSession


async def ask(session: RealtimeSession) -> None:
    await session.send('Has order A0042 shipped yet?')
    await session.wait_for_reply()

Every typed turn costs a backend request, even one that needs no tool, and Live usually says something while the backend works. For small talk that comes out as a filler line ahead of the answer: in our testing, “Thanks, that’s all I needed” got “Oh, okay, let me just check on that. You’re welcome! Have a great day!”

To give Live something to know without running the backend or asking for a reply, send it with respond=False:

from pydantic_ai.realtime import RealtimeSession


async def add_context(session: RealtimeSession) -> None:
    # Not a request to speak: the model takes it into account, and decides for itself whether to mention it.
    await session.send('The caller is a returning guest named Ada.', respond=False)

respond=False does not make the text silent. Live only promises that it doesn’t request speech, and in our testing the model usually acknowledged it out loud anyway (“Welcome back, Ada”). Word it as background (“Internal note: …”) and say in openai_live_instructions what the model should keep to itself.

Live caps the context it takes at 500 tokens. A typed turn of any length still goes to the backend, but one too long for that context (about 496 tokens, with the words that say the user typed it) goes without telling Live what was typed, so Live may leave a terse answer to it unsaid. Context sent with respond=False over the cap raises UserError before anything is sent, so it is not recorded in history; split it, send it as a typed turn, which the backend takes at any length, or give long material to the backend as a tool result instead. enqueue() delivers text the same way once the model is idle, except that system prompts, sent with send() or enqueued on their own, aren’t the user’s words: Live hears them as context as they are (<system>-tagged), and with a reply asked for, the backend gets them as a developer message, so the model doesn’t later tell the caller they said it. Enqueued together with user text, they go out with it as the user’s turn, so enqueue a system prompt on its own when the caller mustn’t be credited with it. The EnqueuedMessagesEvent it produces marks when the text was sent, not when the model took it in. Live speaks to text in the reply it is giving, if it is giving one, rather than in a reply of its own: text sent while the model is speaking is answered by that reply once it says anything more, and wait_for_reply() ends with it.

Live only speaks while its timeline advances, which takes audio, so a session with no microphone, like the text-to-audio example, needs openai_live_idle_audio=True. Without audio, a typed turn still runs the backend and its tools straight away, but Live says nothing, and wait_for_reply() doesn’t return, until audio flows (and in our testing, not always then).

Seeding is text-only in the same spirit: message_history= replays text, transcripts, and thinking text, with tool rounds rendered as readable text because the protocol has nowhere to put function parts. Audio and images in seeded history raise UserError rather than being dropped. Live accepts up to 128 seeded messages and 8,192 tokens in total, so seed a long conversation with its recent end. Within a session, Live manages its own context: once it nears the limit, it continues from a summary of the older conversation, so a long call does not keep every early detail verbatim. The fraction of its context in use, as Live reports it, is context_window_used, which drops once Live summarizes.

Images go to the backend

Live’s voice model sees no images; its delegated backend does. Send one with respond=True, and the backend runs on it straight away: Live opens a delegation, and speaks what the backend makes of the image. The session tracks that like any other reply, so wait_for_reply() waits for it.

from pydantic_ai import BinaryImage
from pydantic_ai.realtime import RealtimeSession


async def show(session: RealtimeSession, photo: bytes) -> None:
    await session.send(BinaryImage(data=photo, media_type='image/jpeg'), respond=True)

An image sent without respond=True raises rather than being added as context. Queued for the backend alone, it goes unseen until the backend next runs, and the voice model, not knowing it exists, answers a question about it without looking. The profile reports this as image_input_requires_response=True.

Usage is measured in seconds

Live bills audio duration, not tokens. The session reports a running total of billable seconds, and Pydantic AI records the increase as audio_seconds on the session’s RunUsage, priced against the Live model, so session.usage.cost includes the call itself once genai-prices knows its rate (see keeping model prices up to date if your data predates gpt-live-1). The value belongs to the session rather than to any one ModelResponse, because Live meters the call as a whole.

Live reports that total only now and then, and last as the session ends, so closing the session asks Live to end it and waits up to two seconds for the final total. A WebRTC sideband’s close() leaves the browser’s call running, so it can’t ask; there, and when Live doesn’t answer in time, the seconds since Live’s last report are missing.

The Live model itself reports no token counts, but the Responses backend it delegates to is billed per token like any other model, and that usage is accumulated with its cache and reasoning breakdowns intact. In a call that delegates, most of the token cost is there.

Those tokens are priced against the backend’s model, not against gpt-live-1, because that is what spent them — so a delegated turn’s cost is right even though the ModelResponse it lands on carries Live’s name. That response records the backend model under delegated_model in its provider_details, and the backend response’s ID under delegated_response_id, so the cost can be recalculated from its usage later, or attributed to the model and response that spent it. (provider_response_id stays unset: Live assigns no ID to the reply it speaks.) The backend’s request is also what a per_request_input_tokens_limit is measured against, since it is the only thing in a Live session that spends input tokens.

For usage limits, that means a cost_limit on UsageLimits bounds the whole call, spoken seconds and backend tokens together, as long as both are priced; token limits bound only the backend. No field caps duration directly, so cap an unpriced call with your own timer or by closing the session.

usage.requests, and the request_limit that bounds it, count the backend’s responses rather than every ModelResponse the session records: on Live most of those are spoken replies, turns whose boundaries are inferred, while the backend’s responses are the requests that spend tokens. The profile says so with responses_are_requests=False. A backend response is counted when its usage arrives, after it has run, so the limit ends the session at the first response past it rather than before that response starts.

Settings

Each family has its own settings class, the realtime counterpart of model run settings, extending the shared settings. Both select the provider voice with openai_voice.

GPT-Live

OpenAILiveModelSettings:

from pydantic_ai.realtime.openai_live import OpenAILiveModel, OpenAILiveModelSettings

settings = OpenAILiveModelSettings(
    openai_voice='marin',
    openai_live_turn_silence_ms=1_500,
    openai_live_store=True,
    openai_live_delegation={'model': 'gpt-6.1-sol', 'verbosity': 'low'},
)
model = OpenAILiveModel('gpt-live-1', settings=settings)

Setting	Purpose
openai_voice	The Live voice, e.g. marin (the provider default). Immutable once the session has started
openai_live_instructions	The Live model’s prompt: persona, language, pacing, style, and when to delegate. The agent’s instructions go to the backend instead
openai_live_delegation	The backend the session delegates to. OpenAILiveResponsesDelegation carries model, extra instructions, reasoning_effort, reasoning_summary, verbosity, max_output_tokens, parallel_tool_calls, and service_tier. reasoning_effort and parallel_tool_calls default to the shared thinking and parallel_tool_calls settings
openai_live_turn_silence_ms	How long the model must stay quiet before the turn boundary is reported. Defaults to 2000
openai_live_store	Whether OpenAI retains the session, which lets it be forked or downloaded later and a dropped connection reconnect with the conversation intact. Defaults to False
openai_live_idle_audio	Stream silence while send_audio() is quiet, so Live speaks without a microphone (its timeline only advances with audio). Live bills silence like speech. Defaults to False
openai_live_data_channel	Which events a WebRTC browser may send and receive over its data channel. Defaults to none, since the sideband runs the session

Voice, audio format, and the starting instructions are fixed for the life of the session, which is why these are session-start settings rather than things to change mid-call. (The Live API can append to the instructions and reconfigure the delegation backend mid-session; Pydantic AI does not expose either yet.) Live exposes no turn-detection, truncation, or token-limit controls: it ignores max_tokens and the TurnDetection knobs, as a standard model ignores a tuning knob it doesn’t support, along with input_transcription_model, since it always transcribes with its own model. It raises for push-to-talk, which it can’t provide. tool_choice reaches the backend when it can’t loop: 'auto', 'none', and ToolOrOutput to limit the tools, applied by trimming the tools the backend is given. 'required' and lists of tool names raise, because the backend applies the choice to every response of a delegation, including the one meant to answer after the tools have run, so it would call tools until a limit ended the session. It has no temperature or other sampling control at all, on either the spoken model or the delegated backend.

gpt-realtime

OpenAIRealtimeModelSettings adds noise reduction, output speed, exact turn detection, and truncation:

from pydantic_ai.realtime.openai import (
    OpenAIRealtimeModel,
    OpenAIRealtimeModelSettings,
)

settings = OpenAIRealtimeModelSettings(
    max_tokens=2_000,
    openai_voice='alloy',
    turn_detection={'sensitivity': 'high', 'silence_duration_ms': 400},
    openai_input_noise_reduction='near_field',
    openai_output_speed=1.1,
    openai_turn_detection={'type': 'semantic_vad', 'eagerness': 'high'},
    openai_truncation={'type': 'retention_ratio', 'retention_ratio': 0.8},
)
model = OpenAIRealtimeModel('gpt-realtime', settings=settings)

openai_turn_detection accepts ServerVAD or SemanticVAD and overrides shared turn_detection. openai_truncation also accepts 'auto' or 'disabled'; retention ratio preserves a stable, cacheable prefix as the session grows. OpenAI realtime does not expose temperature through Pydantic AI.

Pruning finished tool rounds

The API keeps the whole conversation server-side and bills each response for all of it, so a tool-heavy agent pays again for every earlier tool call and result on every later response. Set openai_prune_tool_items to delete a tool round’s items once the model has answered it: when a response completes without calling a tool, the session deletes each answered call together with its output, and any content the tool returned besides it. To keep some tools’ items, for results the model may need to refer back to, name them in openai_keep_tool_items:

from pydantic_ai.realtime.openai import OpenAIRealtimeModel, OpenAIRealtimeModelSettings

model = OpenAIRealtimeModel(
    'gpt-realtime',
    settings=OpenAIRealtimeModelSettings(
        openai_prune_tool_items=True,
        openai_keep_tool_items={'get_customer_profile'},
    ),
)

all_messages() still holds every tool round (see history), and a reconnect’s replay leaves the pruned ones out. The trace records each deletion as a prune tool items span. Azure OpenAI’s GA endpoint supports both settings too; Azure Voice Live ignores them.

Input transcription defaults to 'auto', which uses gpt-live-transcribe; set a supported transcription model ID to pin it or None to disable it. gpt-live-transcribe streams the transcript as the user speaks, but its final transcript can arrive after the reply’s turn is complete. It is still recorded with the user turn it describes, so keep iterating the session if you need it before closing. See Input transcription.

Reasoning

GPT-Live reasons on its delegated backend, so the shared thinking setting (see Thinking) sets the backend model’s reasoning effort, just as it would on a direct request to that model. openai_live_delegation={'reasoning_effort': ...} takes precedence when set.

On gpt-realtime, the shared thinking setting (see Thinking) applies to models whose profile reports supports_thinking, including the gpt-realtime-2 family. True sends reasoning.effort: 'medium' and an effort string selects a level. False sends reasoning.effort: 'none', which turns reasoning off. The GA gpt-realtime ignores the setting.

Neither family streams readable reasoning as ThinkingParts. GPT-Live does record the backend’s encrypted reasoning ahead of each web search, so the history can continue on an OpenAIResponsesModel. To have that reasoning carry readable text, request a summary with openai_live_delegation={'reasoning_summary': ...} ('auto', 'concise', or 'detailed'), as openai_reasoning_summary does on a direct Responses request. The backend summarizes only some of its reasoning, so some of these parts can still be empty.

Browser WebRTC

For browser voice agents, OpenAI recommends WebRTC: the audio flows browser ↔ OpenAI directly, while your backend attaches a control-plane sideband to run the agent. AgentRealtime exposes two signaling helpers, both resolving and binding the agent’s session configuration (instructions, tools, voice, VAD) server-side:

• answer_webrtc_offer — the secure path: relay the browser’s SDP offer to OpenAI from your server, returning the SDP answer and a WebRTCSession to attach a sideband to with agent.realtime(model).session(provider_session=…). The browser never sees a token.
• create_client_secret — mint a short-lived RealtimeClientSecret (ephemeral token) for a browser that negotiates the WebRTC call itself, when you don’t relay the SDP through your backend. gpt-realtime only: GPT-Live has no client secrets.

A GPT-Live session is configured once, by the offer, so history bound with agent.realtime(model, message_history=...) is seeded there, and a sideband opened with different history refuses to attach. The browser’s data channel stays closed unless openai_live_data_channel opens it.

See Deployment for the topology, the secure offer-relay flow, and the sideband trust model, and the realtime WebRTC example for a runnable FastAPI and browser app.

Feature support and limitations

Both families run tool calls asynchronously, so the model keeps talking while a tool runs. See Audio, images, and transcripts, Turns and interruptions, Tools, and Connection lifecycle for the provider-agnostic workflows.

GPT-Live

Feature	Support	Notes
Audio format	Limited parameter support	Mono PCM16 at 24 kHz by default, input and output; 16 kHz by setting both audio_input_sample_rate and audio_output_sample_rate to 16000 through profile=. The API also offers 8 kHz G.711, which Pydantic AI does not expose
Text input	Limited parameter support	Runs the backend as the user’s message; any length (Live is told what was typed up to its 500-token context cap), spoken only while audio flows (see openai_live_idle_audio), and respond=False context possibly spoken anyway
Text output	Unsupported	Live always speaks; read the answer from the transcript on the SpeechPart
Image input	Limited parameter support	For the backend, with respond=True
Manual turns and interruption	Unsupported	Live owns turn-taking and handles barge-in itself, but reports nothing when it does, so a reply the user cut off is recorded as complete, not interrupted. The turn boundary is inferred from silence
Input transcription	Full feature support	Always on in both directions; no model to choose beyond the 'auto' default, and no way to disable it
Input speech events	Unsupported	No speech start/end frames, so a “listening” indicator should read the profile rather than wait for events
Native tools	Limited parameter support	Web search only, run by the backend; sources and citations aren’t recorded
Usage	Limited parameter support	Seconds, not tokens; no duration-based UsageLimits field
Reconnection	Limited parameter support	With a reconnect policy, a session stored with openai_live_store=True is forked, keeping the conversation on OpenAI’s side; otherwise the text of the completed history is replayed into a new session, as much of its recent end as Live’s seeding caps allow. A WebRTC sideband re-attaches to the call’s session instead, keeping the reply and tool calls under way too, unless the drop outlasts the few seconds of events Live sends again

What raises

Live refuses a stated requirement it cannot meet rather than accepting and ignoring it. These raise UserError:

• Before the session connects: turn_detection=False, output_modality='text' (the profile reports supports_text_output=False; see Shared settings), a tool_choice of 'required' or a list of tool names, and a profile= whose audio_input_sample_rate and audio_output_sample_rate keys differ or aren’t 16000 or 24000.
• commit_audio(), clear_audio(), create_response(), and interrupt().
• An image sent without respond=True, respond=False text over the 500-token cap, and seeded history that contains audio or images.
• A WebRTC sideband whose message_history differs from what the offer seeded, including none at all when the offer seeded some, or that sets openai_live_idle_audio.
• create_client_secret(), since Live has no client secrets.

gpt-realtime

Feature	Support	Notes
Audio format	Limited parameter support	Mono PCM16, 24 kHz input and output; the API also offers 8 kHz G.711, which Pydantic AI does not expose
Text output	Full feature support	Select with output_modality='text'
Image input	Full feature support	Images provide context for the next turn
Manual turns and interruption	Full feature support	turn_detection=False plus commit/create verbs; interrupt(played_ms=...) records the heard cutoff
Input transcription	Full feature support	Dedicated model; 'auto' by default
Usage	Full feature support	Token, audio, and cache breakdowns
Native tools	Unsupported	The API offers remote MCP servers, which Pydantic AI does not expose yet (#9032); configure local fallbacks for web capabilities
Reconnection	Full feature support	The connection has no resumable server handle, so Pydantic AI replays completed local history into a new session; in-flight media is lost

Provider-specific quirks

GPT-Live

• Live streams output audio as a continuous track for the whole session (in our recordings, ten frames a second, silent between replies), so an arriving frame says nothing about whether the model is speaking. Pydantic AI drops the idle silence, so stream_audio() yields audio only when the model talks, as on every other provider. Silences of up to half a second inside a reply are forwarded, so a mid-sentence pause doesn’t become a gap in playback; a longer one arrives as a gap.
• Live’s transcript fragments carry their own spacing, except the first fragment of a new segment, which Pydantic AI separates from a sentence that ended the previous one.
• When Live ends the session itself, the reply in progress is recorded as interrupted and the session raises RealtimeError with a code naming the reason: live_session_expired (the duration limit), live_session_content (the safety filter), or live_session_connection_lost. With a reconnect policy, an expired or lost WebSocket session is re-opened instead while the policy’s reconnects last; one the safety filter ended stays ended. On a WebRTC sideband, a lost connection is the browser’s: Live reports a browser that closed its connection as connection_lost, about half a minute later, and the session ends as it would after any other hangup, without an error.
• A delegated backend that fails, stops short, or reports an error is surfaced as a recoverable RealtimeSessionErrorEvent instead: the call goes on. When Live reports such a failure only as a session-level error that names no delegation (for instance Responses handoff incomplete. when the backend runs out of max_output_tokens), Pydantic AI treats the delegated work in flight as the work that failed, with code live_delegation_failed, so the turn still ends. A tool call the failed backend asked for keeps running, and its result is recorded in history, but goes nowhere: the turn doesn’t wait for it.
• A WebRTC sideband that re-attaches after a drop sends again what the dropping sideband may have lost: the tool results of the round under way and, if the backend has said nothing about it within 5 seconds, the request to continue it. Should that go unanswered too, the delegation is given up with a recoverable RealtimeSessionErrorEvent, code live_continuation_lost, so the turn still ends and the call goes on.
• The backend continues a delegated round only once it has every call’s output, and Live has no command to cancel a delegation. When a tool raises, the failure result completes the round as it does on every provider, and the backend carries on (it may try the tool again). When a UsageLimitExceeded stops a call, nothing goes out, so the round doesn’t make another request: Pydantic AI gives it up, and turn boundaries are inferred again.

gpt-realtime

• A response the server can’t generate, for example because its safety system rejected an image in the conversation, arrives as a recoverable RealtimeSessionErrorEvent carrying the provider’s code (such as input_image_safety_violation), followed by an empty response with finish_reason='error' and the error in provider_details['error']. The rejected content stays in the server’s conversation, so every later response fails the same way: seed a new session with history that leaves it out.

Gateway

To route through the Pydantic AI Gateway, use a gateway/-prefixed model string. The gateway does not route GPT-Live yet, so 'gateway/openai:gpt-live-1' fails to connect. Connect through provider='openai' or an OpenAIProvider for now. Once it does, the backend follows the same rule as a direct connection: an agent built on 'gateway/openai:gpt-6-luna' delegates to gpt-6-luna, because both go through the same gateway route.

gpt-realtime routes through the gateway today:

from pydantic_ai import Agent

agent = Agent(instructions='You are a helpful voice assistant.')
realtime = agent.realtime('gateway/openai:gpt-realtime')

Credentials come from gateway_provider. OpenAI-compatible endpoints that expose the realtime protocol can also be supplied through an OpenAIProvider. See Gateway trace propagation.