Skip to content
Docs Pydantic Docs

pydantic_ai.realtime.openai_live

The OpenAI GPT-Live API provider. Requires the openai-realtime optional group (pip install "pydantic-ai-slim[openai-realtime]"), which currently requires openai>=3.26.0. openai 3.12 added Live’s event types.

GPT-Live is a different protocol from the OpenAI Realtime API, not a model served by it, so OpenAILiveModel shares no event mapping with OpenAIRealtimeModel; the model name is what picks between them. The Live model runs the spoken conversation and delegates the work to a backend model configured through OpenAILiveResponsesDelegation in OpenAILiveModelSettings, which receives the agent’s instructions and tools and calls them as ordinary ToolCalls.

Live owns turn-taking entirely, so there is no manual turn control, interruption, or truncation, and no turn-detection setting. It sends no end-of-response frame either: the connection synthesizes ResponseDone after openai_live_turn_silence_ms of silence, and the profile reports synthesizes_turn_boundary=True. Text is delivered as context rather than as a user turn, seeding is text-only, and usage is reported as billable audio seconds instead of tokens. Authentication comes from an OpenAIProvider; Azure OpenAI does not serve Live. See the GPT-Live documentation for the full story.

OpenAI GPT-Live realtime support.

GPT-Live is a separate protocol from the OpenAI Realtime API, not a model served by it, so none of its event mapping is shared: what it borrows from _openai_protocol.py is the parts that are about the provider rather than the protocol (deriving the WebSocket URL, resolving authentication, and mapping handshake failures). The differences that shape the adapter:

  • Audio drives the session. Live has no user-turn event for text: text arrives as context through session.commentary.append (speakable) and session.thinking.append (silent), each capped at 500 tokens. Both are placed on the session’s audio timeline, which only advances while audio flows — so a session whose microphone isn’t streaming defers everything sent to it, unless openai_live_idle_audio has the connection stream silence in the microphone’s place.
  • There is no turn terminal. Live has no response.done equivalent and no transcript-done event; transcripts arrive as timeline fragments. The connection synthesizes ResponseDone once the model has been quiet for openai_live_turn_silence_ms and no delegated work is outstanding, and the profile reports synthesizes_turn_boundary=True so consumers know the boundary is inferred.
  • Work is delegated, not tool-called. The Live model hands a task either to your application (client delegation) or to a Responses backend it drives itself (responses delegation). Only the latter produces typed function calls, so it is what this adapter configures: the agent’s tools are advertised to the backend, its calls arrive nested inside response.event, and their results go back as Responses input items. See the Live docs for the split.
  • Usage has two meters. Live reports its own audio as a cumulative duration in seconds and no tokens at all; the backend it delegates to reports ordinary Responses token usage, which is where most of a call’s token cost is.

OpenAILiveResponsesDelegation

Bases: TypedDict

Settings for the Responses backend a Live session delegates work to.

The Live model runs the conversation; the backend model does the reasoning and calls the agent’s tools. OpenAI’s prompting guide asks for the two prompts to stay separate, and Pydantic AI keeps them separate by construction: the agent’s instructions become the backend prompt, because they describe the work, while live_instructions describes how to speak.

Attributes

model

The Responses model that handles delegated work.

When unset, the backend is, first match wins: the model named after a + in the Live model name ('gpt-live-1+gpt-5.6-sol'); the agent’s own model, when it is an OpenAI model reached the same way as the Live model (directly, or through the same gateway route); then 'auto', which resolves to AUTO_BACKEND_MODEL. There is always a backend, so nothing raises for want of one.

Type: str

instructions

Extra backend instructions, appended after the agent’s own instructions.

Type: str

max_output_tokens

Maximum output tokens per delegated response.

Type: int

parallel_tool_calls

Whether the backend may request several tool calls in one response.

Defaults to the shared parallel_tool_calls setting, since the backend is what calls the tools.

Type: bool

reasoning_effort

Reasoning effort for the backend model.

Takes precedence over the shared thinking setting, which otherwise sets the backend’s effort the way it would on a direct Responses request to that model.

Type: Literal[‘none’, ‘minimal’, ‘low’, ‘medium’, ‘high’, ‘xhigh’]

verbosity

How much detail the backend generates. Does not affect the Live model’s spoken delivery.

Type: Literal[‘low’, ‘medium’, ‘high’]

service_tier

Service tier for delegated Responses requests.

Type: Literal[‘auto’, ‘default’, ‘flex’, ‘priority’]

OpenAILiveModelSettings

Bases: RealtimeModelSettings

Settings for OpenAI GPT-Live sessions.

See RealtimeModelSettings for the shared settings. Live has no turn-detection, truncation, token-limit, or temperature controls: it owns turn-taking entirely, and voice, audio format and instructions are immutable once the session has started.

Attributes

openai_voice

The voice used for Live speech, e.g. marin (the provider default). Immutable after startup.

Type: str

openai_live_instructions

Instructions for the Live model’s spoken behavior: voice, pacing, interruptions, and when to delegate.

The agent’s own instructions describe the work and become the backend prompt, so this is where conversational style belongs. Defaults to a short prompt that tells the model to delegate anything it can’t answer from the conversation itself.

Type: str

openai_live_delegation

Configuration for the Responses backend the Live session delegates work to.

Type: OpenAILiveResponsesDelegation

openai_live_turn_silence_ms

How long the model must stay quiet, in milliseconds, before the session reports the turn complete. Defaults to 2000.

Live has no end-of-turn frame, so this threshold is the turn boundary. Lower it for snappier turn-taking at the risk of ending a turn during a dramatic pause; raise it when replies contain long silences.

Type: int

openai_live_store

Whether OpenAI stores the session so it can later be forked or downloaded. Defaults to False.

With a reconnect policy, a stored session that drops is forked, so the new one has the whole conversation. An unstored one is replaced by a new session seeded with the text of the history so far.

Type: bool

openai_live_idle_audio

Whether to stream silence to Live while the application sends no audio. Defaults to False.

Live’s timeline only advances while it receives audio, so text sent to a session with no microphone streaming waits until audio flows again. With this on, the session streams silence in real time whenever send_audio() has been quiet for a moment, and stops as soon as the application sends audio again, so a text-only session works. Live bills the audio it receives by the second, silence included, so this bills the whole time the session is open.

Type: bool

openai_live_data_channel

Which Live events the browser on a WebRTC call may send and receive over its data channel.

Defaults to none either way. The browser holds only the media: the server’s sideband runs the session, and an open channel would let the browser change the conversation (by appending instructions or context) and read what the sideband sees, starting with the backend’s instructions and tool definitions in session.started. Allow specific events, such as session.output_transcript.delta for live captions, when the browser needs them. The sideband is never restricted, and a WebSocket session has no data channel, so this applies only to answer_webrtc_offer.

Type: DataChannelConfigParam

OpenAILiveConnection

Bases: RealtimeConnection

A live WebSocket connection to the OpenAI GPT-Live API.

Translates Live’s session-timeline events into the shared codec vocabulary, including the two boundaries Live itself never sends: the end of the user’s turn and the end of the model’s reply.

Live streams output audio as a continuous telephony-style track — ten 100 ms frames a second for as long as the session is open, digitally silent when the model isn’t speaking. So a frame arriving means nothing about whether anyone is talking, and voice (a transcript fragment or a non-silent audio frame) is what drives the turn clock.

Attributes

input_transcription_enabled

Live always transcribes both directions; there is no way to turn it off.

Type: bool

reconnect_restores_in_flight_state

Neither way of re-opening a Live session carries on what was in flight.

A fork has the conversation, but not the reply or the delegated work the drop cut off: it has no record of the tool calls that work asked for (checked live). A replacement session has only the history it was seeded with.

Type: bool

Methods

aclose

@async

def aclose() -> None

Cancel the read in flight so closing the socket doesn’t strand its exception.

Returns

None

OpenAILiveModel

Bases: RealtimeModel

OpenAI GPT-Live model.

GPT-Live runs the spoken conversation and delegates the thinking to a Responses backend that Pydantic AI configures with the agent’s instructions and tools, so tools, dependencies, validation, and message history stay on the Pydantic AI side. Reach it through Agent.realtime as 'openai:gpt-live-1', or construct it directly for model-level configuration.

Live differs from the Realtime API in ways that change what a session can do — no text input, an inferred turn boundary, no manual turn control or interruption, and duration-based usage. See the Live docs.

Constructor Parameters

model : OpenAILiveModelName

The model name, e.g. gpt-live-1.

provider : Provider[AsyncOpenAI] | str Default: 'openai'

The provider to use for authentication and the base URL. Defaults to 'openai'.

settings : RealtimeModelSettings | None Default: None

Model settings used as defaults for realtime sessions.

profile : RealtimeModelProfileSpec | None Default: None

Optional override for the realtime model profile.

Attributes

client

The underlying AsyncOpenAI client from the provider.

Type: AsyncOpenAI

Methods

supported_native_tools

@classmethod

def supported_native_tools(cls) -> frozenset[type[AbstractNativeTool]]

Web search, which the delegated backend runs: the only native tool Live lets it have.

Returns

frozenset[type[AbstractNativeTool]]

create_client_secret

@async

def create_client_secret(
    *,
    instructions: str | None = None,
    tools: Sequence[ToolDefinition] | None = None,
    model_settings: RealtimeModelSettings | None = None,
    expires_after_seconds: int | None = None,
) -> RealtimeClientSecret

GPT-Live has no ephemeral client secrets, so this always raises.

Negotiate the call from your server with answer_webrtc_offer instead, so the browser never needs a token.

Returns

RealtimeClientSecret

answer_webrtc_offer

@async

def answer_webrtc_offer(
    sdp_offer: str,
    *,
    instructions: str | None = None,
    tools: Sequence[ToolDefinition] | None = None,
    model_settings: RealtimeModelSettings | None = None,
    message_history: Sequence[ModelMessage] | None = None,
) -> WebRTCAnswer

Start a Live session for a browser’s WebRTC offer, and return the SDP answer and the session to attach to.

Live’s configuration is fixed when the session starts, so everything is set here: the voice and instructions, the delegated backend with the agent’s instructions and tools, and the message_history the call continues, seeded as on a WebSocket session. The browser’s data channel is closed unless openai_live_data_channel opens it. The audio format is negotiated by WebRTC, so the profile’s sample rates don’t apply.

Returns

WebRTCAnswer

connect_webrtc

@async

def connect_webrtc(
    session: RealtimeProviderSession,
    *,
    messages: Sequence[ModelMessage],
    model_settings: RealtimeModelSettings | None,
    model_request_parameters: ModelRequestParameters,
) -> AsyncGenerator[OpenAILiveConnection]

Attach a sideband to a Live session started by answer_webrtc_offer.

The session was fully configured when it started, and Live can’t reconfigure it, so the sideband only runs it: it executes the backend’s tool calls and records the conversation, while the browser holds the audio. For the same reason it seeds nothing: the history it is opened with has to be the history the offer seeded, which it checks against the session Live replays.

Returns

AsyncGenerator[OpenAILiveConnection]

tool_def_to_live 

def tool_def_to_live(tool: ToolDefinition) -> dict[str, Any]

Convert a ToolDefinition to a Live backend function tool.

Returns

dict[str, Any]

seed_input_items 

def seed_input_items(
    messages: Sequence[ModelMessage],
    *,
    provider_name: str,
) -> list[dict[str, Any]]

Map prior history to Live’s startup input list.

Live seeds from text only: user and assistant messages with one text part each. Tool rounds are rendered as readable text because the protocol has no place to put function parts in seeded history. Audio, images, and other media cannot be seeded at all, and the profile says so, which is what makes the session reject them before we get here.

Returns

list[dict[str, Any]]

replay_input_items

@async

def replay_input_items(
    messages: Sequence[ModelMessage],
    *,
    provider_name: str,
) -> list[dict[str, Any]]

Map a session’s own history to the input of the Live session that replaces it after a drop.

Unlike seeding a new session, nothing here is the caller’s to fix, so nothing raises: media is left out, and a long call is cut to its most recent end within Live’s seeding caps rather than refused.

Returns

list[dict[str, Any]]

LatestOpenAILiveModelNames

Latest OpenAI GPT-Live model names.

Default: Literal['gpt-live-1']

OpenAILiveModelName

Possible OpenAI GPT-Live model names.

Since OpenAI supports a variety of date-stamped models, we explicitly list the latest models but allow any name in the type hints.

Default: str | LatestOpenAILiveModelNames

DEFAULT_TURN_SILENCE_MS

How long the model must stay quiet before a turn is considered complete.

Matches the assistant_silence_ms default of OpenAI’s own TranscriptGrouper, which solves the same problem for display transcripts.

Default: 2000

AUTO_BACKEND_MODEL

What a backend model of 'auto' resolves to: the Responses model Pydantic AI currently recommends.

Used when nothing names one: not openai_live_delegation, not the model name, and not the agent. It moves with new OpenAI models, so pin a backend explicitly when its behavior needs to stay put.

Default: 'gpt-6-sol'