OpenAI

Pydantic AI connects an agent to two families of OpenAI voice models, each spoken to over its own protocol by its own model class:

• OpenAILiveModel connects an agent to the GPT-Live family, over the GPT-Live API. The Live model runs the spoken conversation and hands the thinking to a backend model that Pydantic AI configures with the agent’s instructions and tools (see How GPT-Live works).
• OpenAIRealtimeModel connects an agent to OpenAI’s gpt-realtime family of native speech-to-speech models, over the Realtime API. Live is a separate protocol, not a model served by the Realtime API.

You rarely need to choose between the classes yourself: an openai: model name picks the right one (see Model names). Start with the realtime quickstart for the shape of a session, or the text-to-audio example for gpt-realtime.

Setup

To use OpenAI realtime models, install pydantic-ai-slim with the openai-realtime optional group, which bundles the openai package together with the realtime WebSocket transport. GPT-Live’s event types arrived in openai 3.12; the group currently requires openai>=3.26.0:

pip

Terminal

pip install "pydantic-ai-slim[openai-realtime]"

uv

Terminal

uv add "pydantic-ai-slim[openai-realtime]"

Set OPENAI_API_KEY as described in the OpenAI model documentation. Authentication and base URL come from provider, mirroring OpenAIChatModel. The default provider='openai' reads the environment; pass an OpenAIProvider for a custom key or base URL. The realtime WebSocket opens separately, so a custom provider httpx client is not used for it. Sessions run over a server-side WebSocket by default; for browser voice, the browser can exchange media directly over WebRTC while your backend runs the agent (see Connecting a frontend).

Model names

Family	Model class	Example model names	Protocol
GPT-Live	OpenAILiveModel	gpt-live-1	GPT-Live API (/live/sessions)
gpt-realtime	OpenAIRealtimeModel	gpt-realtime, gpt-realtime-2.1, gpt-realtime-2.1-mini	Realtime API (/realtime)

Model availability and aliases can change; use the official OpenAI model documentation as the canonical model list.

The provider prefix alone cannot tell the two protocols apart, so the model name picks the class: an openai: (or gateway/openai:) model name starting with gpt-live routes to OpenAILiveModel, and every other name routes to OpenAIRealtimeModel:

from pydantic_ai import Agent

agent = Agent('openai:gpt-5.6-sol', instructions='You are a helpful voice assistant.')

# GPT-Live, through `OpenAILiveModel`:
live = agent.realtime('openai:gpt-live-1')

# gpt-realtime, through `OpenAIRealtimeModel`:
realtime = agent.realtime('openai:gpt-realtime')

Two cases don’t route by name:

• The azure: prefix always builds an AzureRealtimeModel, whatever the model name. Azure OpenAI does not serve GPT-Live, and passing an azure provider to OpenAILiveModel raises UserError. See the Azure page.
• Constructing a model class directly skips the name routing: the class you construct is the protocol you get. When you instantiate one for model-level configuration, as the settings examples below do, pick the class for the model’s family.

How GPT-Live works

A Live session is more constrained than a gpt-realtime one. It owns turn-taking entirely, takes an image only for its backend to look at, and bills by the second rather than by the token, so read its feature support before porting code between the two.

The backend model

The backend model, the one that does the delegated work, is a second model name. Pydantic AI picks it from the first of these that names one:

1. openai_live_delegation={'model': ...} in the settings.
2. A model named after a + in the realtime model name: 'openai:gpt-live-1+gpt-6-luna' delegates to gpt-6-luna.
3. The agent’s own model, since the backend runs the agent’s instructions and tools. It counts when it is an OpenAI model reached the same way as the Live model, directly or through the same gateway route, so an agent built on 'openai:gpt-5.6-sol' delegates to gpt-5.6-sol. That holds with defer_model_check=True too: the name is resolved when the session connects, as the agent’s own run would resolve it.
4. 'auto': the model Pydantic AI currently recommends (AUTO_BACKEND_MODEL), which moves as OpenAI releases new models.

There is always a backend, so a session never fails for want of one; pin it (1 or 2) when its behavior needs to stay put.

Live accepts any backend name when the session starts and only tries it when it first delegates. A backend model that doesn’t exist, or that your account can’t use, would then fail every delegation, so the first failure ends the session with a RealtimeError (code live_backend_model_unavailable) naming the model, rather than leaving a call that can talk but never look anything up.

from pydantic_ai import Agent

agent = Agent('openai:gpt-5.6-sol', instructions='You look things up.')

# Delegates to the agent's model, gpt-5.6-sol:
realtime = agent.realtime('openai:gpt-live-1')

# Delegates to gpt-6-luna instead:
realtime = agent.realtime('openai:gpt-live-1+gpt-6-luna')

How delegation works

Live splits one agent across two models:

Piece	Where it goes
The agent’s instructions	The backend model, which does the work
The agent’s tools	The backend model, advertised as function tools
Web search	The backend model, as its native web_search tool
openai_live_instructions	The Live model, which does the talking

When the Live model decides it cannot answer from the conversation alone, it opens a delegation and the backend model takes over, calling the agent’s tools as it goes. Those calls arrive as ordinary ToolCalls, so the session runs them through the same tool loop as every other provider, including validation, retries, dependencies, and capability hooks, and sends the results back. Speech and delegated work run independently, so the Live model can keep talking while that happens rather than leaving dead air.

That split is why the agent’s instructions describe the work and openai_live_instructions describes the speech:

from pydantic_ai import Agent
from pydantic_ai.realtime.openai_live import OpenAILiveModelSettings

agent = Agent(
    instructions='You handle order questions for The Terrace. Quote prices exactly as returned.'
)


@agent.tool_plain
async def lookup_order(order_id: str) -> str:
    """Look up an order by its ID."""
    return f'Order {order_id} ships tomorrow.'


realtime = agent.realtime(
    'openai:gpt-live-1',
    model_settings=OpenAILiveModelSettings(
        openai_live_instructions=(
            'Speak warmly and briefly. Say you are checking before you delegate a lookup.'
        ),
        openai_live_delegation={'model': 'gpt-5.6-sol', 'reasoning_effort': 'low'},
    ),
)

Live can also delegate to the client rather than to a backend model, leaving your application to answer in prose. Pydantic AI configures backend delegation, so a client delegation is reported as a RealtimeSessionErrorEvent with code live_client_delegation rather than stalling the call silently.

To run the harder reasoning under your own control instead, expose a tool that delegates to a standard agent; that works here exactly as it does on the other providers.

Live has no command to cancel a delegation, so once the backend has the work it runs to completion, tools included, even if the user takes the question back (“never mind”) in the meantime, and the Live model may still speak its result. Put tools that change state behind approval if a retracted request must not go through.

The turn boundary is inferred

Live sends no end-of-response frame and no transcript-done event, so there is nothing on the wire that marks the end of a reply. The connection infers one: once the model has produced no speech for openai_live_turn_silence_ms and no delegated work is outstanding, it reports the turn complete and the session emits RealtimeTurnCompleteEvent. The end of the user’s turn is inferred the same way.

The profile reports synthesizes_turn_boundary=True so an application can tell an inference from a protocol fact (see Provider support for how to read a profile). Treat the turn boundary as a good guess here: a long dramatic pause can end a turn early, and code that must not act on a partial reply should confirm against the transcript. Lower the threshold for snappier turn-taking, raise it when replies contain long silences.

Delegated work suspends the clock. The model goes quiet while the backend thinks, and ending the turn there would finalize a reply that is still coming.

Text is context, not a user turn

Live has no client event that puts text into the conversation as the user’s own words. Text is delivered as context to the Live model instead, and it relays or answers it:

from pydantic_ai.realtime import RealtimeSession


async def send_context(session: RealtimeSession) -> None:
    # Speakable: the model says this, or something close to it.
    await session.send('Tell the caller their table is ready.')

    # Not a request to speak: the model takes it into account, and decides for itself whether to mention it.
    await session.send('The caller is a returning guest named Ada.', respond=False)

respond=False does not make the text silent. Live only promises that it doesn’t request speech, and in our testing the model usually acknowledged it out loud anyway (“Welcome back, Ada”). Word it as background (“Internal note: …”) and say in openai_live_instructions what the model should keep to itself.

Both forms are capped at 500 tokens by the provider. Longer text raises UserError before anything is sent, so it is neither recorded in history nor waited for by wait_for_reply(); split it, or give long material to the backend as a tool result instead. enqueue() delivers text the same way once the model is idle. The EnqueuedMessagesEvent it produces marks when the text was sent and recorded in history, not when the model took it in. Since Live can take text as context without answering it, bound wait_for_reply() with a timeout rather than relying on a reply always following.

Live’s timeline only advances while it receives audio, so a session with no microphone, like the text-to-audio example, needs openai_live_idle_audio=True.

Seeding is text-only in the same spirit: message_history= replays text, transcripts, and thinking text, with tool rounds rendered as readable text because the protocol has nowhere to put function parts. Audio and images in seeded history raise UserError rather than being dropped. Live accepts up to 128 seeded messages and 8,192 tokens in total, so seed a long conversation with its recent end. Within a session, Live manages its own context: once it nears the limit, it continues from a summary of the older conversation, so a long call does not keep every early detail verbatim. The fraction of its context in use, as Live reports it, is context_window_used, which drops once Live summarizes.

Images go to the backend

Live’s voice model sees no images; its delegated backend does. Send one with respond=True, and the backend runs on it straight away: Live opens a delegation, and speaks what the backend makes of the image. The session tracks that like any other reply, so wait_for_reply() waits for it.

from pydantic_ai import BinaryImage
from pydantic_ai.realtime import RealtimeSession


async def show(session: RealtimeSession, photo: bytes) -> None:
    await session.send(BinaryImage(data=photo, media_type='image/jpeg'), respond=True)

An image sent without respond=True raises rather than being added as context. Queued for the backend alone, it goes unseen until the backend next runs, and the voice model, not knowing it exists, answers a question about it without looking. The profile reports this as image_input_requires_response=True.

Usage is measured in seconds

Live bills audio duration, not tokens. The session reports a running total of billable seconds, and Pydantic AI records the increase as audio_seconds on the session’s RunUsage, priced against the Live model, so session.usage.cost includes the call itself once genai-prices knows its rate (see keeping model prices up to date if your data predates gpt-live-1). The value belongs to the session rather than to any one ModelResponse, because Live meters the call as a whole.

Live reports that total only now and then, and last as the session ends, so closing the session asks Live to end it and waits up to two seconds for the final total. A WebRTC sideband’s close() leaves the browser’s call running, so it can’t ask; there, and when Live doesn’t answer in time, the seconds since Live’s last report are missing.

The Live model itself reports no token counts, but the Responses backend it delegates to is billed per token like any other model, and that usage is accumulated with its cache and reasoning breakdowns intact. In a call that delegates, most of the token cost is there.

Those tokens are priced against the backend’s model, not against gpt-live-1, because that is what spent them — so a delegated turn’s cost is right even though the ModelResponse it lands on carries Live’s name. That response records the backend model under delegated_model in its provider_details, and the backend response’s ID under delegated_response_id, so the cost can be recalculated from its usage later, or attributed to the model and response that spent it. (provider_response_id stays unset: Live assigns no ID to the reply it speaks.) The backend’s request is also what a per_request_input_tokens_limit is measured against, since it is the only thing in a Live session that spends input tokens.

For usage limits, that means a cost_limit on UsageLimits bounds the whole call, spoken seconds and backend tokens together, as long as both are priced; token limits bound only the backend. No field caps duration directly, so cap an unpriced call with your own timer or by closing the session.

usage.requests, and the request_limit that bounds it, count the backend’s responses rather than every ModelResponse the session records: on Live most of those are spoken replies, turns whose boundaries are inferred, while the backend’s responses are the requests that spend tokens. The profile says so with responses_are_requests=False. A backend response is counted when its usage arrives, after it has run, so the limit ends the session at the first response past it rather than before that response starts.

Settings

Each family has its own settings class, the realtime counterpart of model run settings, extending the shared settings. Both select the provider voice with openai_voice.

GPT-Live

OpenAILiveModelSettings:

from pydantic_ai.realtime.openai_live import OpenAILiveModel, OpenAILiveModelSettings

settings = OpenAILiveModelSettings(
    openai_voice='marin',
    openai_live_turn_silence_ms=1_500,
    openai_live_store=True,
    openai_live_delegation={'model': 'gpt-5.6-sol', 'verbosity': 'low'},
)
model = OpenAILiveModel('gpt-live-1', settings=settings)

Setting	Purpose
openai_voice	The Live voice, e.g. marin (the provider default). Immutable once the session has started
openai_live_instructions	How the Live model speaks: pacing, style, and when to delegate
openai_live_delegation	The backend the session delegates to. OpenAILiveResponsesDelegation carries model, extra instructions, reasoning_effort, verbosity, max_output_tokens, parallel_tool_calls, and service_tier. reasoning_effort and parallel_tool_calls default to the shared thinking and parallel_tool_calls settings
openai_live_turn_silence_ms	How long the model must stay quiet before the turn boundary is reported. Defaults to 2000
openai_live_store	Whether OpenAI retains the session, which lets it be forked or downloaded later and a dropped connection reconnect with the conversation intact. Defaults to False
openai_live_idle_audio	Stream silence while send_audio() is quiet, so text lands without a microphone. Live bills silence like speech. Defaults to False
openai_live_data_channel	Which events a WebRTC browser may send and receive over its data channel. Defaults to none, since the sideband runs the session

Voice, audio format, and the starting instructions are fixed for the life of the session, which is why these are session-start settings rather than things to change mid-call. (The Live API can append to the instructions and reconfigure the delegation backend mid-session; Pydantic AI does not expose either yet.) Live exposes no turn-detection, truncation, or token-limit controls, and the shared settings that name them raise rather than being ignored. tool_choice reaches the backend when it can’t loop: 'auto', 'none', and ToolOrOutput to limit the tools, applied by trimming the tools the backend is given. 'required' and lists of tool names raise, because the backend applies the choice to every response of a delegation, including the one meant to answer after the tools have run, so it would call tools until a limit ended the session. It has no temperature or other sampling control at all, on either the spoken model or the delegated backend.

gpt-realtime

OpenAIRealtimeModelSettings adds noise reduction, output speed, exact turn detection, and truncation:

from pydantic_ai.realtime.openai import (
    OpenAIRealtimeModel,
    OpenAIRealtimeModelSettings,
)

settings = OpenAIRealtimeModelSettings(
    max_tokens=2_000,
    openai_voice='alloy',
    turn_detection={'sensitivity': 'high', 'silence_duration_ms': 400},
    openai_input_noise_reduction='near_field',
    openai_output_speed=1.1,
    openai_turn_detection={'type': 'semantic_vad', 'eagerness': 'high'},
    openai_truncation={'type': 'retention_ratio', 'retention_ratio': 0.8},
)
model = OpenAIRealtimeModel('gpt-realtime', settings=settings)

openai_turn_detection accepts ServerVAD or SemanticVAD and overrides shared turn_detection. openai_truncation also accepts 'auto' or 'disabled'; retention ratio preserves a stable, cacheable prefix as the session grows. OpenAI realtime does not expose temperature through Pydantic AI.

Input transcription defaults to 'auto', which uses gpt-live-transcribe; set a supported transcription model ID to pin it or None to disable it. gpt-live-transcribe streams the transcript as the user speaks, but its final transcript can arrive after the reply’s turn is complete. It is still recorded with the user turn it describes, so keep iterating the session if you need it before closing. See Input transcription.

Reasoning

GPT-Live reasons on its delegated backend, so the shared thinking setting (see Thinking) sets the backend model’s reasoning effort, just as it would on a direct request to that model. openai_live_delegation={'reasoning_effort': ...} takes precedence when set.

On gpt-realtime, the shared thinking setting (see Thinking) applies to models whose profile reports supports_thinking, including the gpt-realtime-2 family. True uses the provider default and an effort string selects a level. False sends reasoning.effort: 'none', which turns reasoning off. The GA gpt-realtime ignores the setting.

Neither family surfaces readable reasoning as ThinkingParts. GPT-Live does record the backend’s encrypted reasoning ahead of each web search, so the history can continue on an OpenAIResponsesModel.

Browser WebRTC

For browser voice agents, OpenAI recommends WebRTC: the audio flows browser ↔ OpenAI directly, while your backend attaches a control-plane sideband to run the agent. AgentRealtime exposes two signaling helpers, both resolving and binding the agent’s session configuration (instructions, tools, voice, VAD) server-side:

• answer_webrtc_offer — the secure path: relay the browser’s SDP offer to OpenAI from your server, returning the SDP answer and a WebRTCSession to attach a sideband to with agent.realtime(model).session(provider_session=…). The browser never sees a token.
• create_client_secret — mint a short-lived RealtimeClientSecret (ephemeral token) for a browser that negotiates the WebRTC call itself, when you don’t relay the SDP through your backend. gpt-realtime only: GPT-Live has no client secrets.

A GPT-Live session is configured once, by the offer, so history bound with agent.realtime(model, message_history=...) is seeded there, and a sideband opened with different history refuses to attach. The browser’s data channel stays closed unless openai_live_data_channel opens it.

See Connecting a frontend for the topology, the secure offer-relay flow, and the sideband trust model, and the realtime WebRTC example for a runnable FastAPI and browser app.

Feature support and limitations

Both families run tool calls asynchronously, so the model keeps talking while a tool runs. See Audio, images, and transcripts, Turns and interruptions, Tools, and Connection lifecycle for the provider-agnostic workflows.

GPT-Live

Feature	Support	Notes
Audio format	Limited parameter support	Mono PCM16 at 24 kHz by default, input and output; 16 kHz by setting both audio_input_sample_rate and audio_output_sample_rate to 16000 through profile=. The API also offers 8 kHz G.711, which Pydantic AI does not expose
Text input	Limited parameter support	Context, not a user turn: capped at 500 tokens, delivered only while audio flows (see openai_live_idle_audio), and possibly spoken even with respond=False
Text output	Unsupported	Live always speaks; read the answer from the transcript on the SpeechPart
Image input	Limited parameter support	For the backend, with respond=True
Manual turns and interruption	Unsupported	Live owns turn-taking and handles barge-in itself, but reports nothing when it does, so a reply the user cut off is recorded as complete, not interrupted. The turn boundary is inferred from silence
Input transcription	Full feature support	Always on in both directions; no model to choose and no way to disable it
Input speech events	Unsupported	No speech start/end frames, so a “listening” indicator should read the profile rather than wait for events
Native tools	Limited parameter support	Web search only, run by the backend; sources and citations aren’t recorded
Usage	Limited parameter support	Seconds, not tokens; no duration-based UsageLimits field
Reconnection	Limited parameter support	Over WebSockets; a sideband is not reconnected. With a reconnect policy, a session stored with openai_live_store=True is forked, keeping the conversation on OpenAI’s side; otherwise the text of the completed history is replayed into a new session, as much of its recent end as Live’s seeding caps allow

What raises

Live refuses a stated requirement it cannot meet rather than accepting and ignoring it. These raise UserError:

• Before the session connects: turn_detection, max_tokens, input_transcription_model, output_modality='text' (the profile reports supports_text_output=False; see Shared settings), and a tool_choice of 'required' or a list of tool names.
• commit_audio(), clear_audio(), create_response(), and interrupt().
• An image sent without respond=True, text over the 500-token cap, and seeded history that contains audio or images.
• A WebRTC sideband whose message_history differs from what the offer seeded, including none at all when the offer seeded some.

gpt-realtime

Feature	Support	Notes
Audio format	Limited parameter support	Mono PCM16, 24 kHz input and output; the API also offers 8 kHz G.711, which Pydantic AI does not expose
Text output	Full feature support	Select with output_modality='text'
Image input	Full feature support	Images provide context for the next turn
Manual turns and interruption	Full feature support	turn_detection=False plus commit/create verbs; interrupt(played_ms=...) records the heard cutoff
Input transcription	Full feature support	Dedicated model; 'auto' by default
Usage	Full feature support	Token, audio, and cache breakdowns
Native tools	Unsupported	The API offers remote MCP servers, which Pydantic AI does not expose yet (#9032); configure local fallbacks for web capabilities
Reconnection	Full feature support	The connection has no resumable server handle, so Pydantic AI replays completed local history into a new session; in-flight media is lost

Provider-specific quirks

GPT-Live

• Live streams output audio as a continuous track for the whole session (in our recordings, ten frames a second, silent between replies), so an arriving frame says nothing about whether the model is speaking. Pydantic AI drops the idle silence, so stream_audio() yields audio only when the model talks, as on every other provider. Silences of up to half a second inside a reply are forwarded, so a mid-sentence pause doesn’t become a gap in playback; a longer one arrives as a gap.
• Live’s transcript fragments carry their own spacing, except the first fragment of a new segment, which Pydantic AI separates from a sentence that ended the previous one.
• When Live ends the session itself, the reply in progress is recorded as interrupted and the session raises RealtimeError with a code naming the reason: live_session_expired (the duration limit), live_session_content (the safety filter), or live_session_connection_lost.
• A delegated backend that fails, stops short, or reports an error is surfaced as a recoverable RealtimeSessionErrorEvent instead: the call goes on. When Live reports such a failure only as a session-level error that names no delegation (for instance Responses handoff incomplete. when the backend runs out of max_output_tokens), Pydantic AI treats the delegated work in flight as the work that failed, with code live_delegation_failed, so the turn still ends.

gpt-realtime

• A response the server can’t generate, for example because its safety system rejected an image in the conversation, arrives as a recoverable RealtimeSessionErrorEvent carrying the provider’s code (such as input_image_safety_violation), followed by an empty response with finish_reason='error' and the error in provider_details['error']. The rejected content stays in the server’s conversation, so every later response fails the same way: seed a new session with history that leaves it out.

Gateway

To route through the Pydantic AI Gateway, use a gateway/-prefixed model string. The gateway does not route GPT-Live yet, so 'gateway/openai:gpt-live-1' fails to connect. Connect through provider='openai' or an OpenAIProvider for now. Once it does, the backend follows the same rule as a direct connection: an agent built on 'gateway/openai:gpt-6-luna' delegates to gpt-6-luna, because both go through the same gateway route.

gpt-realtime routes through the gateway today:

from pydantic_ai import Agent

agent = Agent(instructions='You are a helpful voice assistant.')
realtime = agent.realtime('gateway/openai:gpt-realtime')

Credentials come from gateway_provider. OpenAI-compatible endpoints that expose the realtime protocol can also be supplied through an OpenAIProvider. See Gateway trace propagation.