pydantic_ai.realtime
Support for realtime, bidirectional speech-to-speech models (OpenAI Realtime, OpenAI GPT-Live, Azure OpenAI, Gemini Live, xAI Grok Voice, Eleven Realtime, ElevenLabs Agents, and any other provider that streams audio in and out over a persistent connection).
Unlike Model, which is request-response, a realtime model opens a
long-lived connection: you stream audio (or text/images) in, and consume audio, transcripts, and
tool calls as they arrive. The high-level entry point is
Agent.realtime(), followed by
AgentRealtime.session(), which wires the agent’s tools and
instructions into a session and runs the tool loop for you. See the Realtime guide
for a walkthrough.
The flow of a session:
graph LR
App -- "send(): RealtimeSessionInput" --> S[RealtimeSession]
S -- "send(): RealtimeInput" --> C[RealtimeConnection]
M[RealtimeModel] -- "connect()" --> C
C -- "RealtimeFrame" --> S
S -- "RealtimeEvent" --> App
A RealtimeModel opens a
RealtimeConnection (the provider-specific transport).
A RealtimeSession wraps that connection: it translates the
low-level codec events into the shared message/part event vocabulary from
pydantic_ai.messages, builds ordinary
ModelMessage history, and executes tools automatically —
intercepting each ToolCall, running it, sending the
ToolResult back, and emitting a
FunctionToolCallEvent then a
FunctionToolResultEvent. Every tool runs in the
background, so a slow one never blocks the session; whether the model keeps speaking meanwhile is
provider-specific (see Concurrent tool execution).
Provider abstractions & session
| Object | Role |
|---|---|
RealtimeModel | Provider ABC; connect() opens a connection. |
RealtimeModelSettings | Settings shared by realtime providers. |
TurnDetection | Cross-provider automatic VAD sensitivity, padding, and silence configuration. |
KnownRealtimeModelName / infer_realtime_model | Provider-prefixed model IDs and inference. |
RealtimeConnection | Provider ABC; send() content in, frames() of codec and lifecycle events out. |
RealtimeSession | Wraps a connection with automatic concurrent tool dispatch. |
Browser / WebRTC — for browser voice agents, the media flows browser ↔ provider directly while the backend runs a control-plane sideband (OpenAI and Azure OpenAI; see Deployment):
| Object | Role |
|---|---|
RealtimeModel.answer_webrtc_offer | Relay the browser’s SDP offer; return the SDP answer and a WebRTCAnswer / WebRTCSession. |
RealtimeModel.create_client_secret | Mint an ephemeral RealtimeClientSecret for a browser client. |
AgentRealtime.session(provider_session=…) | Attach the sideband session to a RealtimeProviderSession (e.g. a WebRTCSession) and run the agent. |
Inputs — RealtimeSession.send accepts session content
only, in the shared message vocabulary (RealtimeSessionInput):
plain str, image/audio BinaryContent (including
BinaryImage and
BinaryAudio), or a sequence of
these. A string solicits a response, while an image is context-only by default; use respond to
override either behavior. Turn-taking and interruption go through the dedicated
RealtimeSession methods (commit_audio(), clear_audio(),
create_response(), interrupt()), not send(). Use
RealtimeSession.enqueue() to queue text or system
prompt parts for delivery after the active response, or once the model is idle.
Consumption views —
RealtimeSession.stream_audio() yields model
audio chunks ready for playback, while
RealtimeSession.stream_transcripts()
yields finalized speech from both speakers or live deltas with delta=True. These bounded views can
run concurrently with each other and with the session’s raw event iterator.
RealtimeSession.wait_for_reply() waits
until the model has finished the reply it owes, spanning the whole of a tool-calling turn, and runs
alongside an active event iterator rather than competing with it.
RealtimeSession.wait_for_playback()
waits until the single audio view has accounted for all audio emitted so far, whether it was
played or discarded.
RealtimeSession.close() ends the session and every
live view; RealtimeSession.closed exposes its state.
The session starts receiving at whichever comes first: the first outbound call, the first
stream_audio() or stream_transcripts() view, or the first async for over the session. It does
not start on entry, so a view created inside the session block — as every example here does — cannot
miss output the model produced before it subscribed. A fatal receive failure or a tool failure is
raised from active event iteration, which can go on after a tool failure; without an active
iterator, the views end and context exit raises it.
The next outbound session method raises an already-ended receive side’s failure first, and the same
failure is never delivered twice.
The low-level RealtimeConnection.send accepts the
normalized RealtimeInput — a str text turn, a
TextContext item added without soliciting a reply, a raw-PCM
BinaryAudio chunk, or a
BinaryImage frame — which additionally includes the
turn-control verbs (CommitAudio,
ClearAudio,
CreateResponse,
CancelResponse, and
TruncateOutput) that those session methods emit, plus
ToolResult — which the session sends itself as each tool
completes.
Connection events — RealtimeCodecEvent, the low-level codec
vocabulary yielded by a connection:
AudioDelta,
OutputTranscript,
OutputSpeechTimestamps,
InputTranscript,
ToolCall,
ToolCallCancelled,
ResponseDone,
RealtimeInputSpeechStartEvent,
RealtimeInputSpeechEndEvent,
RealtimeOutputSpeechStartEvent,
RealtimeOutputSpeechEndEvent,
RealtimeInputTranscriptionErrorEvent,
RealtimeResponseInterruptedEvent,
RealtimeSessionReconnectEvent,
ConversationCreated,
ConversationItemCreated,
InputRejected,
PartStartEvent,
PartEndEvent,
SessionUsage,
and RealtimeSessionErrorEvent.
Lifecycle events — LifecycleEvent, which a connection
yields with its codec events, each RealtimeFrame from
RealtimeConnection.frames() holding what one provider
message made. They say what the codec events can’t: where each response starts and ends
(ResponseStarted /
ResponseEnded) and which inputs it answers (corrected with
AnswersRevised when the connection learns better), where each
spoken turn starts, joins the conversation, and gets the last of its audio and transcript
(UserTurnStarted /
UserTurnEnded /
UserTurnClosed), where each input joins it
(InputAdded), and which inputs no response will answer
(InputLost /
ResponseRequestRefused), and which tool calls’ results
it won’t answer either (ToolCallsAbandoned). The session
builds history, usage, and wait_for_reply() from them.
Implementing a connection — subclass RealtimeConnection
and implement send() and
frames(). A protocol that identifies its responses,
turns, and inputs (as the OpenAI Realtime API does) is best translated into lifecycle events directly. One that
identifies none of them (as Gemini Live and OpenAI GPT-Live don’t) can hand each input it sends and each
provider message it reads to an InferredLifecycle, which
works the lifecycle out from their order. Either way, a connection that names the output items its codec events
stream (their item_id) can say more about one:
OutputItemDetails keeps what the provider says about it on
the part it makes, and OutputItemCut reports how much of it the
listener heard, which can come after its response ended and revises the copy history recorded. Here, a reply to a
text turn is a response that answers it:
import asyncio
from collections.abc import AsyncIterator, Sequence
from pydantic_ai.realtime.codec import (
OutputTranscript,
RealtimeCodecEvent,
RealtimeConnection,
RealtimeInput,
ResponseDone,
)
from pydantic_ai.realtime.lifecycle import InferredLifecycle, LifecycleEvent, RealtimeFrame
class EchoConnection(RealtimeConnection):
"""Says each text turn back: a stand-in for a protocol that identifies nothing."""
def __init__(self) -> None:
self._lifecycle = InferredLifecycle(transcribes=False)
self._said: asyncio.Queue[str] = asyncio.Queue()
async def send(self, content: RealtimeInput) -> Sequence[LifecycleEvent]:
# Reports the input, and that it failed if writing it raises.
with self._lifecycle.sending(content):
if isinstance(content, str):
self._said.put_nowait(content) # (where a real connection writes to its transport)
# Where it joined the conversation, which the protocol doesn't acknowledge.
return self._lifecycle.take_pending()
def frames(self) -> AsyncIterator[RealtimeFrame]:
return self._lifecycle.frames(self._codec_events())
async def _codec_events(self) -> AsyncIterator[RealtimeCodecEvent]:
while True: # (where a real connection reads its transport)
text = await self._said.get()
yield OutputTranscript(text=text, is_final=True)
yield ResponseDone()
async def main():
connection = EchoConnection()
print(await connection.send('Hello!'))
#> [InputAdded(input_id=0)]
frames = connection.frames()
for _ in range(2):
print([type(event).__name__ for event in await anext(frames)])
#> ['ResponseStarted', 'OutputTranscript']
#> ['ResponseDone', 'ResponseEnded']
frames() ends once the connection is gone: an InferredLifecycle ends what was under way on it then, as
connection_lost() does for a connection
that re-dials. A connection that reads each provider message itself passes its codec events to
frame_for() to get the frame to yield.
The inference assumes what most providers do: the user’s words can be transcribed after the model started
answering them, output audio is the model speaking, and audio the model answers before any transcript of it came
is a spoken turn. A provider that departs from that gets a subclass overriding InferredLifecycle’s extension
points: transcripts_lag_replies,
audio_starts_response,
untranscribed_audio_is_a_turn,
inputs_cut_off_replies,
and takes_turn().
Which inputs a response answers settles each input’s reply obligation, which is what
wait_for_reply() and
RealtimeTurnCompleteEvent wait for, so only the response that
answers an input may name it. A connection that can’t tell yet names its best guess in ResponseStarted.answers
(with basis='inferred') and corrects it with an
AnswersRevised once it can: released hands inputs the response
turned out not to answer to a later one, as InferredLifecycle does for a response an input cut off (with
inputs_cut_off_replies, telling it when the provider heard the user speak with
speech_heard(), where no
RealtimeInputSpeechStartEvent says so), and claimed
settles inputs sent while a response was under way that it answers after all, as GPT-Live does by speaking to text
in the reply it is giving. Neither moves anything in history.
A provider can also give up on tool calls before their results are in: Eleven Realtime does once it starts a reply to
the user, and GPT-Live when its delegated backend fails. The connection then names them in a
ToolCallsAbandoned: the tools keep running and their results
are recorded, but no reply to them is waited for, so the exchange ends with the reply that is under way. Unlike a
ToolCallCancelled, it doesn’t stop them.
Session events — RealtimeEvent, yielded by a
session. The session translates codec events into the shared vocabulary from
pydantic_ai.messages: content streams as
PartStartEvent /
PartDeltaEvent /
PartEndEvent (carrying shared message parts including
SpeechPart, TextPart,
ToolCallPart, and
NativeToolReturnPart), tool execution as
FunctionToolCallEvent /
FunctionToolResultEvent, deferred calls and their inline resolution as
DeferredToolRequestsEvent /
DeferredToolResultsEvent, enqueued-message delivery as
EnqueuedMessagesEvent, events emitted with
RunContext.emit as the
CustomEvent or CapabilityEvent
that was emitted, and the rest as the control-plane events above (RealtimeInputSpeechStartEvent, RealtimeInputSpeechEndEvent,
RealtimeOutputSpeechStartEvent, RealtimeOutputSpeechEndEvent, RealtimeInputTranscriptionErrorEvent,
RealtimeResponseInterruptedEvent, RealtimeSessionReconnectEvent, and RealtimeSessionErrorEvent), plus
RealtimeTurnCompleteEvent, which the
session synthesizes rather than reading off the wire. Usage updates are accumulated on the session and are not yielded.
The lower-level codec vocabulary is documented in
pydantic_ai.realtime.codec, and each provider in its own module:
pydantic_ai.realtime.openai, pydantic_ai.realtime.openai_live,
pydantic_ai.realtime.google, pydantic_ai.realtime.xai,
pydantic_ai.realtime.azure, pydantic_ai.realtime.elevenlabs,
and pydantic_ai.realtime.elevenlabs_agents.
Realtime multimodal session support for bidirectional streaming models.
This package adds support for native speech-to-speech models (OpenAI Realtime, Azure OpenAI,
Gemini Live, xAI Grok Voice, ElevenLabs’ Eleven Realtime, and ElevenLabs Agents) which use a persistent bidirectional connection
rather than the request-response pattern of the standard Model
interface.
The provider-agnostic pieces mirror the request-response layout: pydantic_ai.realtime.model holds
RealtimeModel and model inference,
pydantic_ai.realtime.settings the settings vocabulary, pydantic_ai.realtime.profiles the model
profiles, and pydantic_ai.realtime.codec the low-level
connection vocabulary; concrete providers live in submodules (e.g. pydantic_ai.realtime.openai).
The high-level entry point is Agent.realtime, followed by
AgentRealtime.session.
A session translates the low-level codec events (the connection-facing RealtimeCodecEvent vocabulary)
into the shared message/part event vocabulary from pydantic_ai.messages
(PartStartEvent, FunctionToolCallEvent,
…), plus the realtime control-plane events defined below.
Bases: TypedDict
Describes what a RealtimeModel supports, so a session can tailor its behavior to the model.
Mirrors the shape and supports_-prefixed naming of
ModelProfile for the standard request-response
Model, which realtime models don’t share a hierarchy with.
A RealtimeSession reads these flags to reject unsupported
operations with a clear error before sending them, rather than letting the provider fail
mid-session. Read a model’s via RealtimeModel.profile;
each flag maps to the session methods a provider may not support.
All fields are optional. Consumers treat absent boolean flags as False — except the handful
documented below as defaulting to True, which describe a capability every provider has unless it
says otherwise — absent supported_native_tools as empty, absent sample rates as the values in
DEFAULT_REALTIME_PROFILE, and an absent
context_window as unknown.
Whether the model accepts discrete image/video frames via
image BinaryContent passed to
send.
Type: bool
Whether an image is only taken as the start of a response, rather than as context.
When True, send accepts an image only with
respond=True, and does so without the model supporting general manual turn control; a context-only
image raises. OpenAI GPT-Live sets it: its voice model sees no images, and an image reaches the
delegated backend only when that backend runs on it. Default: False.
Type: bool
Whether the model supports manual turn-taking — commit_audio,
clear_audio, and
create_response (push-to-talk). When False
the model drives turn-taking itself via automatic voice activity detection.
Type: bool
Whether the model supports server-side interruption — cancelling the model’s in-progress response
via interrupt.
Type: bool
Whether the model can truncate its in-progress audio output to the point the user actually heard,
via the played_ms argument of interrupt.
Distinct from supports_interruption:
a provider may support cancelling a response (barge-in) without supporting output truncation. OpenAI
supports both; xAI Grok Voice supports cancellation, but its truncation only lands after a response
has ended, and unreliably, so it reports False.
Type: bool
Whether the model can generate text instead of speech, via
output_modality='text'.
Defaults to True: a realtime model that only speaks is the exception, not the rule. When False,
Agent.realtime rejects output_modality='text' with a
UserError before connecting, rather than letting the provider
fail the handshake (Gemini Live answers 1007 The requested combination of response modalities (TEXT) is not supported by the model) or, worse, silently produce speech anyway (xAI).
Type: bool
Whether the model can generate speech, via
output_modality='audio'.
Defaults to True. A realtime model that only writes text is a perception model rather than a voice
assistant: it takes in a live audio and video stream and answers in text, like Gemini’s
gemini-robotics-er-2-streaming-preview, which a robot’s controller reads rather than a person hears.
When False, Agent.realtime defaults output_modality to 'text'
rather than 'audio', and rejects an explicit output_modality='audio' with a
UserError before connecting (Gemini Live answers 1007 The requested combination of response modalities (AUDIO) is not supported by the model).
Type: bool
Whether the model can seed a session with prior conversation (message_history).
Type: bool
Whether a system prompt sent mid-session goes to the model on its protocol’s channel for the operator’s own text, which it takes as instructions rather than the user’s words.
OpenAI gpt-realtime and Azure send a system message, and OpenAI GPT-Live, with a reply asked for, gives the
delegated backend a developer message. When False (the default), the prompt goes out <system>-tagged, the
way any text would: xAI Grok Voice and Eleven Realtime accept a system message, but their models don’t act
on it. Set it with profile= for a model whose support differs from its provider’s.
Type: bool
Whether the model supports browser WebRTC signaling, ephemeral client secrets, and a server-side
control-plane sideband via answer_webrtc_offer,
create_client_secret, and
connect_webrtc.
Supported by OpenAI and Azure OpenAI. OpenAI GPT-Live supports the offer relay and the sideband but has no client secrets. Gemini Live and xAI Grok Voice are WebSocket-only.
Type: bool
Whether prior images can be included when seeding a session with message_history.
Type: bool
Whether retained user audio can be included when seeding a session with message_history.
Type: bool
Whether the model supports reasoning/thinking configuration via the
thinking setting — OpenAI’s gpt-realtime-2*
reasoning models, the Gemini Live models except Vertex AI’s gemini-live-2.5-flash and gemini-3.8-live
(its gemini-3.8-live-extended-thinking variant always thinks), xAI’s grok-voice-latest and
grok-voice-think-* models, and OpenAI GPT-Live, whose delegated backend does the reasoning. When
False (the default), a thinking setting is silently ignored rather than sent to a model that
would reject it.
Type: bool
Whether the model keeps the conversation going while a tool call runs. Default: 'never'.
With async tool calls, the model can keep speaking (typically saying what it’s doing) and answer the user while a tool runs, and the result reaches it when it’s ready. Without them, the model goes quiet until the result is back.
'optional' models run tool calls asynchronously when the session asks for it with the
async_tool_calls setting, and
'never' and 'always' models ignore that setting.
OpenAI Realtime, Azure OpenAI, OpenAI GPT-Live, xAI Grok Voice, and
gemini-3.8-live-extended-thinking are 'always'; the Gemini native-audio models and
gemini-3.8-live are 'optional'; other Gemini Live models are 'never', since they accept an
async declaration and then wait for the result anyway.
Type: AsyncToolCallMode
Deprecated: use async_tool_call_mode instead.
A resolved RealtimeModel.profile still carries it,
as async_tool_call_mode != 'never'. Setting it in a profile= override is translated (with a
deprecation warning): True to 'optional' and False to 'never', except that it never changes
an 'always' model, which it never could.
Type: bool
Whether the model natively renders a tool’s return_schema
(Gemini Live’s function-declaration response schema). Where it can’t, a tool that opted in via
include_return_schema gets the schema injected into its description instead, exactly as on a
standard Model.
Type: bool
The native tools the model runs server-side, e.g.
WebSearchTool.
Agent.realtime validates the session’s native
tools against this set before connecting, raising a UserError
that names any the model doesn’t support — mirroring the classic
Model.supported_native_tools check.
Type: frozenset[type[AbstractNativeTool]]
Whether the provider reports when the user starts and stops speaking, as
RealtimeInputSpeechStartEvent and
RealtimeInputSpeechEndEvent.
emits_ rather than supports_ because this describes events that appear in the stream, not an
operation the session can invoke. The OpenAI-protocol providers (OpenAI, Azure OpenAI, xAI) emit
them; Gemini Live does not — a UI that shows a “listening” indicator should read this flag rather
than wait for events that will never arrive.
Type: bool
How finely the model reports when its speech is heard, with the
output_speech_timestamps setting on:
per 'word', per 'segment' of one or more words, or None (the default) when it doesn’t report it.
Azure AI Voice Live and ElevenLabs Agents report words.
Type: SpeechTimestampGranularity | None
Whether RealtimeTurnCompleteEvent is inferred
by Pydantic AI rather than reported by the provider.
synthesizes_ rather than supports_ for the same reason as
emits_input_speech_events:
it describes how a stream is produced, not an operation the session can invoke.
False (the default) means the provider sends an explicit end-of-response frame, so the turn
boundary is a protocol fact. True means the protocol has no such frame and the adapter derives
the boundary from output timing, so it is a good guess rather than a guarantee: a long pause
mid-sentence can end a turn early, and an application that must not act on a partial reply should
confirm against the transcript. OpenAI GPT-Live is the only model that sets it.
Type: bool
Whether each response the session records is one request to the model, for usage.requests and
the request_limit that bounds it.
True (the default) counts every recorded ModelResponse, as a
standard run does, and checks the limit before each response starts. False is for a model whose
recorded responses aren’t what it is asked for: OpenAI GPT-Live’s spoken replies are turns the
session infers, while the requests that spend tokens go to the backend it delegates to. There, each
response-scoped usage report counts as one request, and the limit is checked as each arrives, so the
report that would go past it ends the session.
Type: bool
Whether a response’s reported token usage covers everything the model holds in context.
RealtimeSession.context_window_used
divides the latest response’s total_tokens by
context_window when the provider
doesn’t report the fraction itself, which is only meaningful when those tokens are the whole
context. Defaults to True. False makes it None instead of a misleading ratio: xAI Grok Voice
reports only the input a response added, and OpenAI GPT-Live’s token usage belongs to the
backend it delegates to rather than to the voice model’s own context.
Type: bool
The sample rate, in Hz, expected for raw PCM audio input.
Read it via RealtimeSession.audio_input_sample_rate
(or RealtimeModel.audio_input_sample_rate
before a session exists), which fall back to the default when a profile omits it.
Type: int
The sample rate, in Hz, produced in raw PCM audio output deltas.
Read it via RealtimeSession.audio_output_sample_rate
(or RealtimeModel.audio_output_sample_rate
before a session exists), which fall back to the default when a profile omits it.
Type: int
The maximum number of tokens the model can hold in a session, input and output combined. Default: None (unknown).
When no profile layer sets this, RealtimeModel.profile
fills it in from genai-prices data if the model is known
there. Set it explicitly for models it doesn’t know, e.g. profile={'context_window': 128_000}.
Read it via RealtimeModel.context_window.
Bases: TypedDict
Cross-provider automatic voice-activity detection (VAD) knobs.
Set as RealtimeModelSettings.turn_detection to turn
automatic detection on with these settings. (Pass True for the provider defaults, or False to
disable it entirely for push-to-talk.) Each field maps to the closest knob on each provider; for
finer, provider-specific control use the provider-prefixed escape hatch (openai_turn_detection,
xai_turn_detection, google_vad), which fully overrides this when set.
How readily the provider detects turn boundaries (speech start/end). Higher is snappier but more
prone to false triggers. Defaults to the provider default. Maps per provider: OpenAI / Azure / xAI → server-VAD threshold
(low≈0.7, medium≈0.5, high≈0.3); Gemini → both start and end sensitivity (low→low,
high→high, medium leaves the provider default); Eleven Realtime → semantic-VAD eagerness.
Type: Literal[‘low’, ‘medium’, ‘high’]
Audio retained before detected speech onset, in milliseconds. Honored by OpenAI, xAI, Gemini, and Eleven Realtime. Defaults to the provider default.
Type: int
Silence required to mark the end of speech, in milliseconds. Honored by OpenAI, xAI, Gemini, and Eleven Realtime. Defaults to the provider default.
Type: int
Bases: ModelAPIError
A realtime connection or protocol failure: the session could not be opened, or is over.
Raised when the handshake fails, the provider closes the session, a send fails, or
reconnecting gives up. A rejected WebSocket upgrade is the
exception: it carries an HTTP status, so it raises
ModelHTTPError like a regular request.
A subclass of ModelAPIError, since losing the connection
to a realtime provider is the same kind of failure as a request-response call that couldn’t reach
the API. Catch it specifically to separate the session’s own failures from those of any text agent
the session delegates to.
Bases: TypedDict
Settings to configure a realtime model session.
Defines the common settings vocabulary used across realtime model providers. A provider treats a
setting it can’t apply as a standard model does: tuning knobs like
max_tokens are ignored silently, so the same settings work across providers;
parallel_tool_calls=False emits an UnsupportedSettingWarning;
and a tool_choice that forces a tool call raises UserError before
connecting. A few settings a provider can’t honor raise too: see each field’s docs and the
provider pages.
Providers with additional generation parameters extend it, e.g.
GoogleRealtimeModelSettings.
The maximum number of tokens to generate per response before stopping.
Supported by: OpenAI, Azure OpenAI, Gemini, and Eleven Realtime. xAI accepts it but ignores it, and OpenAI GPT-Live and ElevenLabs Agents ignore it too.
Type: int
Whether to allow parallel tool calls.
Supported by: OpenAI, Azure OpenAI, Azure AI Voice Live (where True leaves the model’s default,
since some models can’t call tools in parallel), and OpenAI GPT-Live, where it applies to the
delegated backend unless openai_live_delegation sets its own parallel_tool_calls. xAI accepts it
but ignores it, and Gemini, Eleven Realtime, and ElevenLabs Agents ignore it too: on those, False with
function tools emits an UnsupportedSettingWarning, as
on a standard model.
Type: bool
Whether the model keeps the conversation going while a tool call runs. None (the default) leaves it to the model.
With async tool calls, the model can keep speaking (typically saying what it’s doing) and answer the user while a tool runs, and the result reaches it when it’s ready. Without them, the model goes quiet until the result is back. This pays off for tools that take a noticeable moment; see Concurrent tool execution for the tradeoffs.
Only models whose profile’s
async_tool_call_mode is
'optional' offer a choice, and every one of them defaults to off. The others ignore this setting,
since they either always or never run tool calls asynchronously. A model that always does warns about an
ignored False (with function tools) with an
UnsupportedSettingWarning, as a standard model does
for a guarantee it can’t keep.
Supported by: the Gemini native-audio models and gemini-3.8-live.
Control which function tools the model can use.
See the Tool Choice guide for detailed documentation. Every
form is resolved exactly as it is for a standard run, including the error a name that matches no
tool raises; a session has no output tools, so
ToolOrOutput restricts the function tools while leaving the
model free to just speak.
It is applied once, when the session is created, and holds for every response. 'none' and
function-tool allow-lists are enforced on every provider by restricting the tools advertised to the
model. Every provider raises UserError before connecting for a
choice that forces a tool call ('required' or a list of tool names): applied to every response,
including the one after a tool result, it would never let the model answer, and Gemini can’t express
it at all. Use ToolOrOutput to restrict the tools instead. ElevenLabs Agents restricts a hosted
agent’s attached tools through its toggle-gated tool_ids conversation override.
Supported by: OpenAI, Azure OpenAI, Gemini, xAI, Eleven Realtime, ElevenLabs Agents, and OpenAI GPT-Live,
for 'auto', 'none', and ToolOrOutput restrictions.
Type: ToolChoice
Model used to transcribe the user’s audio input, so their turns are captured into history.
'auto' (the default) uses the provider’s recommended realtime transcription model; pass a
specific id (e.g. 'gpt-live-transcribe') to pin one, or None to disable transcription (see
audio_retention to retain the raw audio instead).
None turns transcription off on every provider that can disable it. OpenAI GPT-Live, Eleven Realtime,
and ElevenLabs Agents always transcribe, so on them None leaves transcription on.
A pinned id applies only to the providers that
transcribe with a separate model, and the others ignore it, as a standard model ignores a tuning knob it
doesn’t support: Gemini transcribes natively (google_input_transcription configures its own
transcription), and OpenAI GPT-Live, Eleven Realtime, and ElevenLabs Agents transcribe with their own model.
Supported by: OpenAI, Azure OpenAI, Gemini (None only), and xAI.
Type: KnownRealtimeTranscriptionModelName | str | None
The single modality generated by the model. Defaults to 'audio', or to 'text' for a model whose
profile reports supports_audio_output=False.
Unlike most settings here, an unsupported value is not silently ignored: a model whose
profile reports supports_text_output=False
(or supports_audio_output=False, for 'audio') raises UserError
before connecting, because a session that quietly spoke instead of writing would be worse than one that
didn’t start.
Text is supported by: OpenAI, Azure OpenAI, Vertex AI’s half-cascade gemini-live-2.5-flash, Gemini’s
text-only gemini-robotics-er-* streaming models, and ElevenLabs Agents (via the hosted agent’s
toggle-gated conversation.text_only override). The other Gemini Live models and xAI always generate
audio: read the spoken answer from the transcript on the SpeechPart
instead.
Type: Literal[‘audio’, ‘text’]
Whether to report when each piece of the model’s speech is heard in its audio, so an application can line
the text up with playback (for example, to reveal words as they are spoken). Defaults to False.
When on, the timestamps stream as SpeechPartDelta.timestamps
and are recorded on the SpeechPart in history. How fine they are is the
model profile’s
output_speech_timestamp_granularity;
a model that reports none emits an UnsupportedSettingWarning
when the session opens.
Supported by: Azure AI Voice Live (per word, with an Azure voice), ElevenLabs Agents (per word).
Type: bool
Enable or configure reasoning/thinking, mirroring the unified
thinking setting on the request-response models.
True enables it at effort 'medium' on OpenAI (including GPT-Live’s backend), Azure OpenAI, and
Azure AI Voice Live, at the closest supported thinking level to 'medium' on Gemini, and at 'high',
its only enabled effort, on xAI. 'minimal'/'low'/'medium'/'high'/'xhigh'/'max' selects an
effort level. False disables thinking (sent as effort 'none' on OpenAI, Azure OpenAI, Azure AI
Voice Live, and xAI).
OpenAI and Gemini apply it only to models whose profile reports
supports_thinking. Other models
silently ignore it. Providers with a richer native config expose it separately
(e.g. Gemini’s google_thinking_config), which takes precedence.
Supported by: OpenAI gpt-realtime-2* models (also on Azure), reasoning chat models like gpt-5
on Azure AI Voice Live, the Gemini Live models except Vertex AI’s gemini-live-2.5-flash and
gemini-3.8-live (its gemini-3.8-live-extended-thinking variant always thinks), xAI’s reasoning Grok
Voice models (grok-voice-latest and the grok-voice-think-* family), and OpenAI GPT-Live, where it
sets the reasoning effort of the delegated backend model if that model reasons, unless
openai_live_delegation sets its own reasoning_effort.
Type: ThinkingLevel
Automatic voice-activity detection (VAD) / turn-taking. Modeled on
thinking:
- Absent (the default) or
True: automatic turn detection on, at the provider’s defaults. False: disable it — push-to-talk, drive turns manually withcommit_audio()/create_response()(only on providers whose model profile reportssupports_manual_turn_control).TurnDetection: on, with specific cross-provider knobs.
For finer, provider-specific control use the provider-prefixed setting documented on each provider’s
settings type (openai_turn_detection, xai_turn_detection, google_vad); when present it fully
overrides this field.
OpenAI GPT-Live and ElevenLabs Agents own turn-taking server-side: they ignore the TurnDetection
knobs, and raise UserError for False, as Gemini does.
Type: bool | TurnDetection
Seconds to wait for the realtime protocol handshake to complete. Defaults to 30.0.
On OpenAI, Azure OpenAI, xAI, and Eleven Realtime this bounds the wait for each handshake event; on Gemini it bounds opening the socket and waiting for the session setup to complete; on ElevenLabs Agents it bounds the wait for the conversation initiation metadata.
Supported by: OpenAI, Azure OpenAI, Gemini, xAI, Eleven Realtime, and ElevenLabs Agents.
Type: float
ReconnectPolicy to transparently recover from a
dropped connection. Without a policy, an unexpectedly closed connection is fatal: the low-level
connection reports a non-recoverable session error and RealtimeSession raises
RealtimeError from iteration.
What server-side state survives a reconnect depends on the provider (see
ReconnectPolicy). Setting a policy enables Gemini’s
native session resumption unless google_enable_session_resumption=False is set explicitly — that
combination raises
UserError at connect time, since a re-dial without
resumption would lose the conversation.
Supported by: OpenAI, Azure OpenAI, Gemini, xAI, Eleven Realtime, and OpenAI GPT-Live, which forks a session stored
with openai_live_store=True and otherwise replays the local history into a new one. ElevenLabs Agents
can’t resume a conversation, so it ignores a policy with an
UnsupportedSettingWarning, and a drop ends the session.
Type: ReconnectPolicy
An ephemeral client secret (short-lived token) that a browser can use to talk to a provider directly.
Minted server-side by create_client_secret
so a long-lived API key never reaches the browser. The token is bound to a session configuration
(instructions, tools, voice, VAD) and expires quickly (OpenAI: about a minute).
The ephemeral secret to hand to the browser client.
Kept out of the repr so logging or inspecting the object doesn’t leak the live token into logs.
Type: str Default: field(repr=False)
When the secret expires (timezone-aware UTC).
Type: datetime
Where the browser posts its WebRTC SDP offer, with the secret as its bearer token.
The answer comes back as the response body, and the call’s id in its Location header. On Azure OpenAI the
URL carries webrtcfilter=on, so the browser’s data channel gets only the events that are safe for it to see,
as on a call answered with answer_webrtc_offer.
None when the provider takes no WebRTC offers.
Type: str | None Default: None
Raw provider fields returned alongside the secret (e.g. the resolved session object).
Kept out of the repr alongside value: the resolved session carries the instructions and tool
definitions, so a logged repr would otherwise expose them. Access the field explicitly to read it.
Type: dict[str, Any] | None Default: field(default=None, repr=False)
Bases: Protocol
A handle to a provider-side realtime session that a server sideband connection can attach to.
The transport-neutral contract that Agent.realtime accepts as
provider_session: it needs only the owning provider (to check the attaching model matches) and an
opaque session identifier (to address the control-plane connection). Different transports satisfy it
with their own handle types — a WebRTC HTTP relay yields a WebRTCSession;
a provider that negotiates over a WebSocket can supply its own.
The provider that owns the session (e.g. 'openai' or 'azure'); must match the model attaching to it.
Type: str
The provider-assigned identifier used to address the session’s control-plane connection.
Type: str
A RealtimeProviderSession for a WebRTC call.
Produced by answer_webrtc_offer and
passed as provider_session to Agent.realtime to run the
agent loop over the call’s control plane while the browser owns the audio.
The provider that owns the call (e.g. 'openai' or 'azure'); must match the model attaching to it.
Type: str
The provider-assigned call identifier (OpenAI/Azure return it as the call_id in the Location header).
Type: str
Raw provider details about the call (e.g. the original Location header).
Type: dict[str, Any] | None Default: None
Alias for session_id under the OpenAI/Azure wire name.
Type: str
The provider’s WebRTC SDP answer plus the WebRTCSession to attach to.
Return sdp to the browser to complete the WebRTC handshake, then pass session as provider_session
to Agent.realtime to run the sideband session.
The provider’s SDP answer, to send back to the browser as the remote description.
Type: str
The call handle the server sideband session attaches to.
Type: WebRTCSession
One incremental transcript update, carrying everything needed to render it.
Yielded by RealtimeSession.stream_transcripts(delta=True).
A realtime session is duplex, so both speakers’ transcripts stream at the same time and a caption
UI needs to know not just what was said but which turn to put it in — otherwise two
consecutive turns by the same speaker run together.
Identifies the turn this update belongs to, stable for the life of the session.
Use it as the key for whatever you render a turn into: every update with the same index belongs
to the same speech part.
Type: int
Who is speaking.
Type: Literal[‘user’, ‘assistant’]
The text this update added, when it added any.
Empty when the provider revised the turn instead of extending it — speech recognition is
revisable, and a correction can’t be expressed as an addition. Render transcript and this never
matters.
Type: str
The full transcript of this turn so far.
Render this, keyed on index, and captions are correct whatever the provider does: no
accumulating, no special case for a revision, and a dropped update (if a consumer fell behind)
self-corrects on the next one.
Type: str
Bases: AbstractModel
Abstract base class for realtime model providers.
RealtimeModel and the request-response
Model share AbstractModel.
A realtime model opens a persistent bidirectional connection for streaming content in and out.
Like Model, the settings attribute and the model_settings
passed to connect are typed as the shared RealtimeModelSettings;
each provider narrows to its own TypedDict subclass internally with a cast (as the
request-response models do for ModelSettings), rather than the base class being generic over the
settings type.
The shared RealtimeModelSettings fields this model class ignores.
Works like Model.unsupported_settings: when a session sets one
that changes what it guarantees (parallel_tool_calls=False with function tools, or a reconnect policy),
opening it emits an UnsupportedSettingWarning; the tuning
knobs among them, like max_tokens, are ignored silently. Empty by default, so a model class that declares
nothing never warns. tool_choice, thinking and async_tool_calls aren’t listed here: a forced tool_choice
raises on every realtime model, and the other two are resolved through the profile. Nor is a setting a class
honors in part, like Gemini’s input_transcription_model, whose None turns transcription off.
Type: frozenset[RealtimeModelSettingName] Default: frozenset()
Model settings used as defaults for realtime sessions.
Type: RealtimeModelSettings | None Default: settings
The model name, e.g. gpt-realtime.
Type: str
The provider API base URL, when this model is backed by a provider.
The realtime model profile.
Resolution order mirrors Model.profile (later layers
override earlier ones):
DEFAULT_REALTIME_PROFILE— base values for every key.- The provider’s
realtime_model_profile(model_name)result — provider-specific defaults. - A best-effort
context_windowvalue from genai-prices, unless the provider or a partial user profile explicitly set the field (including toNone). - The model class’s adjustments for what this instance supports beyond its name, such as the API surface its client talks to (e.g. a flag verified on one surface only).
- The user’s
profile=argument — a partial dict merged on top, OR a callable(resolved) -> profilefor full control.
Then supported_native_tools is intersected with what this model class actually implements, so
the resolved profile is the single source of truth for what is usable, and the deprecated
supports_async_tool_calls
is derived from
async_tool_call_mode.
Type: RealtimeModelProfile
The resolved profile’s context_window.
The sample rate, in Hz, expected for raw PCM audio input.
Also available on the session as
RealtimeSession.audio_input_sample_rate;
read it here when audio capture must be configured before a session exists.
Type: int
The sample rate, in Hz, of the raw PCM audio the model produces.
Also available on the session as
RealtimeSession.audio_output_sample_rate;
read it here when audio playback must be configured before a session exists.
Type: int
def __init__(
*,
settings: RealtimeModelSettings | None = None,
profile: RealtimeModelProfileSpec | None = None,
) -> None
Store the model-level settings and profile= layers, as Model’s constructor does.
Also loads genai-prices’ data snapshot, which profile
consults for context_window, so that one-time cost stays off the event loop.
@classmethod
def supported_native_tools(cls) -> frozenset[type[AbstractNativeTool]]
Return the native tool types implemented by this realtime model class.
frozenset[type[AbstractNativeTool]]
def unsupported_native_tool_settings(native_tool: AbstractNativeTool) -> frozenset[str]
Return the names of the settings set on native_tool that this model can’t apply.
Works like Model.unsupported_native_tool_settings:
a native tool with any of these is treated as unsupported for the session. The default is an empty set.
@abstractmethod
def connect(
*,
messages: Sequence[ModelMessage],
model_settings: RealtimeModelSettings | None,
model_request_parameters: ModelRequestParameters,
) -> AbstractAsyncContextManager[RealtimeConnection]
Open a connection to the realtime model.
AbstractAsyncContextManager[RealtimeConnection] — An async context manager yielding a RealtimeConnection.
messages : Sequence[ModelMessage]
Prior conversation and the current request carrying session instructions,
projected to the provider’s initial conversation items. Replayable text, transcripts,
thinking, tool rounds, images, and retained user audio are seeded according to the
model profile; content the provider cannot represent raises UserError.
model_settings : RealtimeModelSettings | None
Optional provider-specific settings.
Function and native tools available to the session.
def connect_webrtc(
session: RealtimeProviderSession,
*,
messages: Sequence[ModelMessage],
model_settings: RealtimeModelSettings | None,
model_request_parameters: ModelRequestParameters,
) -> AbstractAsyncContextManager[RealtimeConnection]
Attach a control-plane (sideband) connection to an existing provider-side session.
The returned connection runs the agent loop over the session’s control channel while the browser
exchanges audio with the provider directly, so the sideband doesn’t own the audio transport. Only
realtime models whose provider supports WebRTC server-side controls (OpenAI and Azure OpenAI)
implement this; the default raises UserError and points
callers to the WebSocket transport.
AbstractAsyncContextManager[RealtimeConnection]
@async
def create_client_secret(
*,
instructions: str | None = None,
tools: Sequence[ToolDefinition] | None = None,
model_settings: RealtimeModelSettings | None = None,
expires_after_seconds: int | None = None,
) -> RealtimeClientSecret
Mint an ephemeral RealtimeClientSecret for a browser client.
Binds the token to the given session configuration so a browser can open a realtime connection
directly without ever holding a long-lived API key. Only implemented by providers that support
ephemeral tokens (OpenAI and Azure OpenAI); the default raises
UserError and points callers to the WebSocket transport.
RealtimeClientSecret
@async
def answer_webrtc_offer(
sdp_offer: str,
*,
instructions: str | None = None,
tools: Sequence[ToolDefinition] | None = None,
model_settings: RealtimeModelSettings | None = None,
message_history: Sequence[ModelMessage] | None = None,
) -> WebRTCAnswer
Relay a browser’s WebRTC SDP offer to the provider and return the SDP answer plus a WebRTCSession.
This is the secure signaling path: the server (holding the API key) negotiates the WebRTC call
on the browser’s behalf, so the browser never sees a token. Return
WebRTCAnswer.sdp to the browser, then pass
WebRTCAnswer.session as provider_session to
Agent.realtime. Only implemented by
providers that support WebRTC (OpenAI and Azure OpenAI); the default raises
UserError and points callers to the WebSocket transport.
message_history is the conversation the call continues. A model whose protocol only takes history
when a session starts (OpenAI GPT-Live) seeds it into the session this creates; the others seed it
when the sideband attaches, from the history that session is opened with, so pass the same history
to both.
WebRTCAnswer
@async
def hang_up(session: RealtimeProviderSession) -> None
End a provider-side call, such as a browser’s WebRTC call, for everyone on it.
Takes the WebRTCAnswer.session that
answer_webrtc_offer returned, so a server
can end a call with or without a sideband attached to it; a sideband session can use
RealtimeSession.hang_up instead. A call that has
already ended is not an error; the provider likely answers the same for a call negotiated with a
different API key or project, so that reads as already ended too. Only implemented by realtime
models whose provider can end a WebRTC call from the server (OpenAI gpt-realtime and GPT-Live, and Azure
OpenAI); the default raises UserError.
Bases: TypedDict
How to recover when a realtime connection drops mid-session.
Set as the reconnect key of RealtimeModelSettings,
either as a model-level default (settings=) or per session (model_settings=).
On a dropped connection the session is re-dialed and its configuration (instructions, tools,
voice, …) re-applied, emitting a
RealtimeSessionReconnectEvent event. What server-side state
survives depends on the provider: OpenAI Realtime, Azure OpenAI and xAI Grok Voice have the session
replay its finalized message history into the new connection (prior turns survive; buffered input
audio and a reply in flight do not), while Gemini Live restores prior turns through native session
resumption, enabled automatically whenever a reconnect policy is set
(Gemini honors an explicit google_enable_session_resumption=False opt-out by refusing the
combination with a UserError).
Number of re-dial attempts per drop before giving up and raising
RealtimeError. Defaults to 3.
Type: int
Total successful reconnects allowed for the life of the session.
max_attempts bounds the retries for a single drop, and resets once a dial succeeds, so on its
own it cannot stop a session that reconnects, drops, and reconnects forever. This bounds the whole
session instead.
The default is generous for the case this exists to serve: providers end sessions at a duration
cap (OpenAI at 60 minutes) and a long-running session legitimately renews at that boundary, so 50
covers days of continuous conversation. It only bites a server that hangs up as fast as we dial.
Defaults to 50.
Type: int
Base backoff delay in seconds; doubles each attempt up to max_delay. Defaults to 0.5.
Type: float
Maximum backoff delay in seconds. Defaults to 30.0.
Type: float
Whether to apply random jitter to each backoff delay to avoid thundering herds. Defaults to True.
Type: bool
Wraps a RealtimeConnection, building message history and auto-executing tools.
The session translates the connection’s low-level codec events into the shared message/part event
vocabulary from pydantic_ai.messages and accumulates ordinary
ModelMessage history as the conversation proceeds, so a
session can hand off to Agent.run via
all_messages:
- assistant speech becomes
PartStartEvent/PartDeltaEvent/PartEndEventevents carrying aSpeechPart(speaker='assistant'), finalized into aModelResponseat the end of the turn; - user speech becomes the same part events with
speaker='user', finalized into aModelRequest; - a tool call becomes a
ToolCallPart(start/end) plus aFunctionToolCallEventwhen execution starts and aFunctionToolResultEventcarrying a normalizedToolReturnPartwhen it settles.
Tools always run concurrently with the session. The session keeps streaming events while a tool runs, so the model can keep speaking and user speech keeps being processed, then sends the result back over the connection once it is ready. This mirrors how a person can keep talking while work happens.
Tool outcomes use the same normalized history shapes as a classic agent run: retries become
outcome='retried' returns, denials retain their outcome, and structured returns preserve
content, metadata, and typed tool_kind identity. Realtime
tool-output channels are string-only, so the structured part is rendered only when it is sent.
OpenAI-protocol connections send additional user content as a follow-up conversation item; Gemini
includes a text fallback in its tool response. If the provider cancels an in-flight call, the
session records a synthetic interrupted return for valid history but does not send that abandoned
result to the provider.
History is accumulated in the order events are reported. Provider item IDs keep interleaved input
transcripts associated with their correct user turns; providers without item IDs retain
arrival-order association. Tool results are the exception: a tool’s
FunctionToolResultEvent streams whenever the tool
finishes (possibly after later turns), but in all_messages()
its result part is placed directly after the response carrying its call — request-response APIs
require that adjacency, so the history stays valid for a handoff to a standard
Agent.run.
Images and video frames streamed with send are
stored as ordinary user image turns by default. Set retain_images_every_n above 1 to sample
high-rate frame streams, and retain_images_max (default 100) to bound how many stay in
history — the oldest retained image is evicted first, so a long-running stream can’t grow memory
without limit. Audio kept by audio_retention is bounded the same way: retain_audio_max_seconds
(default 1800, 30 minutes across both speakers) caps how much stays in memory, evicting the
oldest retained audio first while keeping its transcript.
When constructing a session directly, use it as an async context manager. The context owns the
receive pump, background tool tasks, and instrumentation spans; iteration only reads its event
queue. AgentRealtime.session enters the session
before yielding it, so the usual agent API remains a single async with block.
Cumulative token usage and tool-call counts for the session, updated as events stream in.
Pass usage to Agent.realtime to accumulate
into a shared RunUsage; otherwise a fresh one is used.
Default: usage if usage is not None else RunUsage()
Whether the session has been closed.
Type: bool
The final result once the session context has exited, otherwise None.
Type: AgentRunResult[str] | None
What the connected model supports, as RealtimeModel.profile.
Available here because the session is what a call actually holds: agent.realtime() accepts a
model name and builds the model itself, leaving nothing else to read the profile from. The
audio sample rates have their own dedicated properties —
audio_input_sample_rate and
audio_output_sample_rate —
so most code never needs to read the profile directly.
Type: RealtimeModelProfile
The sample rate, in Hz, of the raw PCM audio this session expects.
Resample the microphone to this rate before
send_audio: audio sent at the wrong rate
is heard as a chipmunk (or slow-motion voice) rather than reported as an error.
Type: int
The sample rate, in Hz, of the raw PCM audio stream_audio() yields.
Play output at this rate; it can differ from
audio_input_sample_rate
(Gemini Live, for example, listens at 16 kHz and speaks at 24 kHz).
Type: int
Fraction of the model’s context window occupied, as of the latest report.
When the provider reports it (OpenAI GPT-Live), this is the latest value it reported. Otherwise
it is computed as for a standard run’s
RunContext.context_window_used: the latest
response’s total_tokens over the model’s
context_window.
The value can go down as the session continues, when the provider compacts or truncates the
conversation server-side. Returns None when the ratio cannot be calculated: when the context
window or usage is unknown, before the first response, or when the model’s response usage
doesn’t measure what the context holds (see
response_usage_covers_context).
The playback position of the session’s single stream_audio() consumer, in raw PCM bytes.
A chunk counts as played once the consumer comes back for the next one, so under the
documented playback pattern — write each chunk to the device and wait until it is consumed
before pulling the next — this is the position at the last chunk boundary the device
reached, the same value a hand-rolled counter incremented after each write would report.
The chunk being written is not counted until it is done, so barge-in attributes at most one
chunk less than was really heard, never more. A consumer that instead buffers ahead reads
too far; count actual device consumption yourself in that case. Made to be passed to
interrupt(played_bytes=...) on
barge-in. Requires exactly one active stream_audio() iterator, like that call; once the
connection has failed and ended that iterator, reading this raises the failure instead.
Type: int
This session’s Conversation, ready to hand to a text run.
The same bundle AgentRunResult.conversation
produces, so a spoken conversation can be continued by
Agent.run and handed back again, carrying its usage
rather than restarting the budget each time it changes modality.
A session opened without a conversation_id resolves one here the way
Agent.realtime does up front — continuing the
conversation its seeded history belongs to, or starting a new one — and keeps it, so every
bundle taken from the session and every message it records shares one identity to store the
conversation under.
Type: Conversation
@async
def close() -> None
Close the session and end its live stream views.
This method is idempotent. Concurrent callers wait for the same teardown, which continues if
a caller is cancelled; leaving the session context waits for it to finish. Active
stream_audio() and
stream_transcripts() iterators
finish cleanly, with any buffered items discarded. The surrounding model context owns the
underlying connection, so it remains open until that context exits.
A tool can call this method through
ctx.realtime_session to end the session. The
calling tool does not resume, and its call is recorded as interrupted. On a WebRTC sideband this only
detaches, and the browser’s call stays up: use
hang_up() to end it.
Raises a failure that hasn’t been delivered yet: whatever ended the session (a provider hangup,
an exceeded usage_limits), or a failed tool that nothing was iterating to receive. A failure
already raised by event iteration or an outbound session method is not raised again.
@async
def hang_up() -> None
End the call, then close the session.
On a WebRTC sideband session,
close() only detaches this server from the call,
and the browser stays connected to the provider (and billed) until it hangs up itself. This asks the
provider to end the call for everyone. On a session that owns its connection, closing it already
ends the call, so this is the same as close(). Either way, the session is closed afterwards, as
by close().
A tool can call this method through
ctx.realtime_session, as it can close(): the
call is still ended after the tool itself is cancelled by the teardown.
Raises ModelHTTPError or
ModelAPIError if the provider refuses to end the call,
and UserError if the model can’t hang up WebRTC calls.
def stream_audio() -> AsyncIterator[bytes]
Stream model audio chunks ready for playback.
The iterator contains only live model audio, in playback order. It never repeats retained
audio from finalized speech parts. On a WebRTC sideband the browser owns the audio path, so
this raises UserError; consume the browser’s remote media
track instead.
The subscription starts when this method is called, so audio the model produces between
the call and the consumer’s first iteration is buffered rather than missed. A view handed
to asyncio.create_task therefore misses nothing while waiting for its first turn on the
event loop. Models generate speech several times faster than it plays, so each iterator
buffers up to five minutes of audio: a consumer that plays each chunk before pulling the next
receives a long reply in full. Past that bound (a consumer that stopped iterating, or never
started), the oldest chunk is dropped so audio playback cannot stall tool execution, turn
tracking, or the main event stream; an unconsumed view keeps that bounded buffer until it
is collected. A device-paced consumer also feeds
played_audio_bytes, and on
barge-in interrupt(played_bytes=...)
discards buffered chunks the user will never hear, so a playback loop can iterate one
stream for the whole session. Closing the session discards buffered chunks and ends the
iterator cleanly. If a tool fails while the event stream has no consumer, this view ends and
the error is raised when the session closes.
@async
def wait_for_playback() -> None
Wait until the session’s single audio view has accounted for all audio emitted so far.
Call this after a reply finishes generating, before closing the session or opening the
microphone, so buffered audio is not cut off. Playback advances with the same one-chunk lag
as played_audio_bytes: a chunk
counts once the consumer requests the next one. Audio the view never plays counts as accounted for
rather than played: discarded by a barge-in or by the view’s buffer overflowing, or emitted before
the view subscribed. The wait also ends if the view is closed or abandoned, or the session closes.
Requires exactly one active
stream_audio() iterator; once the
connection has failed and ended that iterator, this raises the failure instead.
@async
def wait_for_reply() -> None
Wait until the model has finished the reply it owes, if any.
Returns once the exchange in progress reaches the boundary
RealtimeTurnCompleteEvent marks — after a
tool-calling turn, that is the answer that follows the tool results, not the response that
called them. A reply is the response that answers what was sent, not one that was already under
way when it reached the provider (see The turn boundary).
Returns immediately when the model owes nothing, so a reply that finished between
the send() and this call is not waited for twice
over; it also returns if the session closes or ends (a response passed a usage limit), if the
exchange fails so the reply can no longer come (a tool’s result would pass a usage limit, or a
failed tool’s result could not be sent), or if the provider refuses the request for the reply
(reported as a RealtimeSessionErrorEvent).
A tool that raises doesn’t end the exchange: the model is told the call failed, and this waits
for its answer (see Tools).
This is the wait async for event in session would otherwise be written out to perform, and
unlike that loop it can run while something else is iterating the session, so a caller that
only wants the turn boundary does not have to take over the event stream to find it:
from pydantic_ai import Agent
agent = Agent(instructions='You are a helpful voice assistant.')
async def main():
async with agent.realtime('openai:gpt-realtime').session() as session:
await session.send('Say hello.')
await session.wait_for_reply()
The reply can show up in all_messages() later than
it ends (by up to 30 seconds), where history waits for the transcript of a spoken turn before it (see
History).
Waiting for the model to stop generating is not the same as waiting for the speaker to stop
playing: pair it with
wait_for_playback() before opening
the microphone, so the reply is not cut off and the model does not hear itself.
def stream_transcripts(*, delta: Literal[False] = False) -> AsyncIterator[SpeechPart]
def stream_transcripts(*, delta: Literal[True]) -> AsyncIterator[TranscriptUpdate]
Stream speech transcripts for both the user and assistant.
By default, yields finalized SpeechPart instances — one
per completed turn, carrying its speaker and full transcript. A model output item whose
transcript grows after its part ended, possibly while a later item is still streaming, is
yielded again each time it grows, with the transcript so far; delta=True carries the
index to replace it by.
Pass delta=True for live captions, which yields
TranscriptUpdates carrying the new text, the turn’s
full transcript so far, the speaker, and an index identifying the turn. Both speakers
stream at once, so that index is what lets a UI keep two turns apart instead of running
them together. Empty updates and finalized parts without a transcript are omitted.
Final transcripts and deltas are separate subscriptions, so one never crowds out the other. The subscription starts when this method is called, so nothing said between the call and the consumer’s first iteration is missed. Each iterator buffers up to 512 items; if its consumer falls behind (or never starts), the oldest is dropped, because captioning must not be able to stall tool execution, turn tracking, or the main event stream. Closing the session discards buffered items and ends the iterator cleanly; an unconsumed view otherwise keeps its bounded buffer until it is collected. If a tool fails while the event stream has no consumer, this view ends and the error is raised when the session closes.
AsyncIterator[SpeechPart | TranscriptUpdate]
def all_messages() -> list[ModelMessage]
A snapshot of the seeded history plus messages recorded during this session.
Returns a copy, so the result doesn’t change as the session continues. Feed it into
Agent.run(message_history=...) to hand the
conversation off to a standard agent run. Images streamed with send() are recorded according
to retain_images_every_n, bounded by retain_images_max, and audio retained by
audio_retention is bounded by retain_audio_max_seconds.
def new_messages() -> list[ModelMessage]
A snapshot of the messages created during this session (excluding the seeded history).
@async
def send(
content: RealtimeSessionInput | Sequence[RealtimeSessionInput],
*,
respond: bool | None = None,
) -> None
Feed content into the session.
A str is a complete text turn that the model replies to. Do not follow send('...') with
create_response(): that asks for two responses. An image is added as context without
soliciting a response, while audio is streamed into the input buffer.
A SystemPromptPart is a system prompt added
mid-session, such as a note from the application about what just happened: it goes out on the
provider’s channel for the operator’s own text where there is one (a system message on OpenAI
and Azure), and as <system>-tagged user text otherwise, and is recorded in history as the
SystemPromptPart it is. Like text, it solicits a response unless respond=False.
Set respond=False to add text, a system prompt, or an image as context without soliciting a
response. Set respond=True to solicit a response to text, a system prompt, or an image; this
requires manual turn control for images. respond=True cannot be used with audio. For a sequence, an explicit respond value
adds every item except the last as context and applies that value to the last item. With the
default respond=None, each item keeps its usual behavior.
Accepts the shared message vocabulary: plain text as a str, a SystemPromptPart, image/audio
BinaryContent (including
BinaryImage and
BinaryAudio), or a sequence of these inputs, dispatched
in order. Text, system prompts, and retained images are recorded in session history; audio is recorded later
through its transcript and/or audio_retention. retain_images_every_n=1 records every image,
while larger values keep the first image and then one of every N; retain_images_max bounds
how many stay recorded, evicting the oldest first. Sending an image is gated on the model
profile’s image-input support and raises UserError when it is unsupported.
send() accepts session content only. Turn-control verbs (CommitAudio, ClearAudio,
CreateResponse, CancelResponse, TruncateOutput) are driven through the dedicated methods
(commit_audio(), clear_audio(), create_response(), interrupt()), and
ToolResult is sent by the session itself as each tool
completes (see _execute_tool) — neither is accepted here.
def enqueue(
*content: EnqueueContent,
priority: PendingMessagePriority = 'asap',
) -> str | None
Enqueue content to be delivered to the model when the session is ready for it.
Use this from code driving the session. For tools, use
RunContext.enqueue. Unlike
send(), which is delivered immediately even
while the model is speaking, enqueued content waits for the appropriate turn boundary and
triggers a model response when delivered.
str | None — The enqueue_id of the queued message, or None when there was nothing to enqueue.
One or more EnqueueContent items. Realtime
delivery supports text (a str or TextContent)
and SystemPromptParts, which are joined into
one turn. System prompts alone are delivered and recorded as
send(SystemPromptPart(...)) delivers and
records them. Next to text, they’re joined into the user turn using the
<system>…</system> convention to indicate that the text is not the person speaking.
Either way, that marks where it came from, not that it should be handled silently: the
model still gets a turn and decides whether to reply, call a tool, or move on. To add
context without prompting a turn at all, use
send(..., respond=False). A turn with
text is recorded in history as a UserPromptPart.
Calling with no positional arguments is a no-op. A message holding TextContent and
only user prompt parts is recorded as those parts instead, so their metadata, which
isn’t sent, stays in history.
When to deliver:
'asap' (default) — as soon as no reply is being generated; a reply already in
progress is allowed to finish first.
'when_idle' — only once the model is idle: no response is in flight and all
'asap' items have been delivered.
UserError— If the session has not been entered or is closed, or the content cannot be delivered over the live input channel.
@async
def send_audio(data: bytes | AsyncIterable[bytes]) -> None
Stream mono PCM16 audio to the model: a single chunk, or an async iterable of chunks.
Given an async iterable — a microphone stream, a WebSocket receive loop — each chunk is
forwarded as it arrives and the call returns when the iterable is exhausted, so a whole
capture loop can be one task: asyncio.create_task(session.send_audio(microphone)). If the
session is closed while consuming that iterable, the call returns cleanly at the next chunk
the source yields. Cancel the task in application code if the source can stall indefinitely.
Passing a single chunk to a session that is already closed still raises
UserError.
Resample the audio to
audio_input_sample_rate
first (24 kHz on the OpenAI-protocol providers, 16 kHz on Gemini):
raw bytes carry no rate, so the wrong one is heard as a chipmunk rather than reported.
With a reconnect policy, a chunk sent
while a dropped connection is being re-dialed is discarded rather than raised, so a microphone
task survives the reconnect. That includes a one-shot clip passed as a single chunk: resend it
after the
RealtimeSessionReconnectEvent if it matters.
@async
def commit_audio() -> None
Commit buffered input audio as a user turn (manual turn-taking / push-to-talk).
@async
def clear_audio() -> None
Discard buffered, uncommitted input audio.
@async
def create_response() -> None
Ask the model to respond now (manual turn-taking, after commit_audio).
If a response is already in flight, the request is held until it completes and is dropped if the user barges in. Returning does not mean the model has started speaking.
@async
def interrupt(*, played_ms: int | None = None) -> None
def interrupt(*, played_bytes: int) -> bool
Barge-in: cancel the model’s in-progress response, optionally truncating its audio first.
This is server-side only — it stops generation and (when a playback position is given) syncs
the provider’s transcript to what was actually heard. Generation outruns playback, so the reply
being heard has often finished generating: with played_bytes, the session still truncates it,
and history records it as interrupted where playback stopped (see
Barge-in).
With played_ms, the caller owns all playback accounting: flushing locally buffered
playback is the caller’s responsibility, and deciding whether to interrupt at all is too.
With played_bytes, the session owns the rest: it attributes the position to the current
turn (adjusting for audio the stream dropped or that predates the subscription), discards
buffered audio the user will never hear — chunks already queued, and the cancelled
response’s stragglers as they arrive — then truncates and cancels, returning True. The
cancel is left out while the provider has a speech segment open on a model whose own turn
detection stops the response being spoken over, where a client cancel racing it can be
applied to the next response; an interruption raised outside a speech segment always
cancels. On a
model without output truncation the unheard audio is still flushed and the response
cancelled; only the provider-side transcript sync is unavailable. When everything emitted
was already played there is nothing to cut off: a reply that has not reached its first audio
chunk is cancelled untruncated, since it is about to speak over the user, and otherwise
nothing is sent and False is returned — reporting an interruption anyway would make the
provider discard part of a completed reply. This form needs exactly one
stream_audio() iterator, kept for the
session’s lifetime; with several (or none) the position is ambiguous and a
UserError is raised — keep your own accounting and
pass played_ms instead.
Playback position in milliseconds from the start of the model’s current audio output. When given, the model’s output audio and its transcript are truncated to this point before the response is cancelled.
Playback position as the total raw PCM bytes actually played from the
session’s single stream_audio() iterator. Mutually exclusive with played_ms.
@async
def __aiter__() -> AsyncIterator[RealtimeEvent]
Read translated events from the session queue without owning session resources.
AsyncIterator[RealtimeEvent]
The model exchange is over: what RealtimeSession.wait_for_reply() waits for has come.
It is synthesized by the session when a response ends and the model owes nothing more: no reply to an
input sent (text, a tool’s result, a request for a response), no answer to a tool call, and no response
still under way. It also ends an exchange a failure ended (a tool’s result would pass a usage limit, or a failed
tool’s result couldn’t be sent), once. Input transcription can finish after this event.
On WebRTC sidebands, provider playback can also continue until
RealtimeOutputSpeechEndEvent.
Event type identifier, used as a discriminator.
Type: Literal[‘realtime_turn_complete’] Default: 'realtime_turn_complete'
The provider detected that the user started speaking.
Useful for barge-in: stop playing any buffered model audio when this arrives, since the model’s in-progress turn is being interrupted.
Reported by OpenAI, Azure OpenAI, and xAI. Gemini Live does not report speech onset.
Provider id of the user input item this speech segment belongs to, when reported.
Type: str | None Default: None
Event type identifier, used as a discriminator.
Type: Literal[‘realtime_input_speech_start’] Default: 'realtime_input_speech_start'
The provider cut the model’s in-progress response short.
Arrives as soon as the provider interrupts, ahead of its response terminal, so it’s the point at which to flush buffered model audio.
Reported by Gemini Live, which interrupts server-side when it hears the user speak. The other
providers report the user’s speech onset as
RealtimeInputSpeechStartEvent and leave the cancellation
to interrupt, so they never report this.
Event type identifier, used as a discriminator.
Type: Literal[‘realtime_response_interrupted’] Default: 'realtime_response_interrupted'
The provider detected that the user stopped speaking.
Useful as a ‘processing’ indicator: the user’s turn has ended and the model is about to respond.
Provider id of the user input item this speech segment belongs to, when reported.
Used to attach retained input audio (audio_retention='input_audio'/'all') to the right user turn
when turns overlap, since transcripts for different items can finalize out of order.
Type: str | None Default: None
Event type identifier, used as a discriminator.
Type: Literal[‘realtime_input_speech_end’] Default: 'realtime_input_speech_end'
The provider started playing the model’s audio to the listener.
Only reported where the provider, rather than your code, holds the audio on its way to the listener: on a WebRTC sideband the media flows browser ↔ provider, so the session never sees audio and this is its only signal that the model has become audible. An ordinary session owns the audio and knows when it starts playing it, so no provider reports this there.
This is about playback, not generation: the provider produces audio faster than it plays it, so this can arrive well after the audio itself was generated.
Event type identifier, used as a discriminator.
Type: Literal[‘realtime_output_speech_start’] Default: 'realtime_output_speech_start'
The provider stopped playing the model’s audio to the listener.
The counterpart to
RealtimeOutputSpeechStartEvent, and the
honest end of a spoken turn: because the provider generates audio far ahead of playing it, it is
still talking long after
RealtimeTurnCompleteEvent reports the response
finished. Drive a “speaking” indicator from this pair rather than from turn completion.
Event type identifier, used as a discriminator.
Type: Literal[‘realtime_output_speech_end’] Default: 'realtime_output_speech_end'
The provider failed to transcribe a user audio input turn, but the session continues.
This is recoverable; item_id and content_index locate the affected user turn.
Human-readable error message.
Type: str
Provider error category, if any.
Type: str | None Default: None
Provider error code, if any.
Type: str | None Default: None
Provider conversation-item ID for the affected user turn, when available.
Type: str | None Default: None
Content index within the affected user turn, when available.
Type: int | None Default: None
Event type identifier, used as a discriminator.
Type: Literal[‘realtime_input_transcription_error’] Default: 'realtime_input_transcription_error'
The connection dropped and was automatically re-established; inspect state_restored for continuity.
Session configuration (instructions, tools, voice, …) is restored on every reconnect. Conversation state is restored either by the provider’s native session resumption (Gemini Live when enabled, xAI Grok Voice, a stored OpenAI GPT-Live session) or by the session replaying its local history into the fresh server-side conversation (OpenAI/Azure OpenAI, an unstored GPT-Live session).
Whether the reconnect carried the conversation through without cutting a turn off, regardless of mechanism — native provider resumption or a local-history replay.
True means nothing in flight was lost: the provider either resumed the in-flight response itself
(Gemini Live, xAI Grok Voice) or there was no turn in progress when the connection dropped.
False means a turn the drop interrupted was settled before continuing — its partial reply is
recorded as an interrupted response and any running tool calls as cancelled returns — so
all_messages() stays a coherent history.
Finalized turns from before the drop survive where the provider restores them (the OpenAI/Azure
OpenAI local replay) and are lost where it does not; either way, treat the interrupted turn as over
and expect the model to stay quiet until the next input.
Type: bool Default: False
Event type identifier, used as a discriminator.
Type: Literal[‘realtime_session_reconnect’] Default: 'realtime_session_reconnect'
A provider-reported error occurred in the session.
Human-readable error message.
Type: str
Provider error category, e.g. invalid_request_error or server_error.
It can also be a category Pydantic AI defines for a condition it detects itself:
'tool_calls_abandoned' means Eleven Realtime abandoned tool calls whose results hadn’t been sent yet, so
those results won’t reach the model.
Type: str | None Default: None
Provider error code, if any.
Type: str | None Default: None
Whether the session can continue. A protocol error is recoverable; a dropped connection is not.
Type: bool Default: True
Event type identifier, used as a discriminator.
Type: Literal[‘realtime_session_error’] Default: 'realtime_session_error'
def infer_realtime_model(
model: KnownRealtimeModelName | str,
provider_factory: Callable[[str], Provider[Any]] | None = None,
) -> RealtimeModel
Infer a realtime model from a provider:model identifier.
The provider is one of openai, azure, xai, elevenlabs, google (the Gemini Developer API),
google-cloud (Vertex AI), or elevenlabs-agents (whose “model name” is an agent id, e.g.
elevenlabs-agents:agent_0101...) — e.g. openai:gpt-realtime — or a
Pydantic AI Gateway route (gateway/openai:gpt-realtime,
gateway/google:gemini-live-2.5-flash), which connects through the gateway’s built-in provider —
the provider string is passed to the realtime model as its provider, so authentication and the
base URL come from gateway_provider.
RealtimeModel
model : KnownRealtimeModelName | str
The realtime model identifier, in the format provider:model.
Function that instantiates the provider, called with the provider prefix
exactly as written ('azure', 'gateway/google', …) — the counterpart of the same
parameter on infer_model, for credentials that don’t
come from the environment. The provider it returns is used as-is. When omitted, each
realtime model resolves the prefix itself (e.g. azure through
AzureProvider.for_realtime()).
How much audio a RealtimeSession retains in its history.
Values other than 'transcript_only' are additive: transcripts are always kept, and the named audio is
retained alongside them.
'transcript_only'(default): keep only transcripts; drop all audio bytes.'input_audio': also retain the user’s spoken audio.'output_audio': also retain the model’s spoken audio.'all': retain both sides’ audio.
Retained audio is stored on the SpeechPart’s audio as WAV
BinaryContent. Live audio deltas remain raw PCM. Retained
input audio is split only at boundaries the provider reports. OpenAI, Azure, and xAI report the end
of each detected speech segment, so each user part normally contains audio sent since the preceding
speech-end boundary through the current one. This can include inter-turn microphone input. Gemini
does not report speech-end boundaries, so its user part contains everything sent since the preceding
response completed through the current response completion, including silence sent while the model
is responding. Retention records the microphone stream only; it does not mix the model’s output audio
into the user’s part unless that output is present in the microphone input itself.
How much retained audio a session keeps is bounded by retain_audio_max_seconds on
AgentRealtime.session; the oldest is evicted first, keeping
its transcript.
Default: TypeAliasType('AudioRetention', Literal['transcript_only', 'input_audio', 'output_audio', 'all'])
Whether a realtime model keeps the conversation going while a tool call runs.
'never': the model waits for the tool’s result before it says anything else.'optional': it’s up to the session, via theasync_tool_callssetting.'always': the model keeps talking, and answering the user, while a tool runs; it has no mode that waits.
Default: TypeAliasType('AsyncToolCallMode', Literal['never', 'optional', 'always'])
The content types a caller feeds into RealtimeSession.send.
Session content only, in the shared message vocabulary: a str is a complete text turn,
BinaryContent carries an image frame, WAV audio (unwrapped to
raw PCM before it is streamed, matching the history-seeding path), or a raw PCM chunk
(media_type='audio/pcm'), and a SystemPromptPart is a
system prompt added mid-session, delivered on the provider’s channel for the operator’s own text. The session normalizes these before forwarding them to the connection.
Turn-control verbs (CommitAudio, ClearAudio, CreateResponse, CancelResponse,
TruncateOutput) are connection-level vocabulary driven through the dedicated RealtimeSession
methods (commit_audio(), clear_audio(), create_response(), interrupt()), and
ToolResult is sent by the session itself when a tool
completes — neither is accepted by send().
Default: TypeAliasType('RealtimeSessionInput', 'str | BinaryContent | SystemPromptPart')
Union of events yielded by RealtimeSession.
This is a strict subset of AgentStreamEvent.
Content is streamed as the shared PartStartEvent /
PartDeltaEvent / PartEndEvent
events (carrying SpeechParts and
other shared message parts, including TextPart,
ToolCallPart, and
NativeToolReturnPart), tool execution as
FunctionToolCallEvent /
FunctionToolResultEvent, deferred calls and their resolution
as DeferredToolRequestsEvent /
DeferredToolResultsEvent, enqueued-message delivery
as EnqueuedMessagesEvent, events emitted with
RunContext.emit as the
CustomEvent or
CapabilityEvent that was emitted, and the rest as realtime
control-plane events.
Default: TypeAliasType('RealtimeEvent', PartStartEvent | PartDeltaEvent | PartEndEvent | FunctionToolCallEvent | FunctionToolResultEvent | DeferredToolRequestsEvent | DeferredToolResultsEvent | EnqueuedMessagesEvent | CustomEvent | CapabilityEvent | RealtimeTurnCompleteEvent | RealtimeInputSpeechStartEvent | RealtimeResponseInterruptedEvent | RealtimeInputSpeechEndEvent | RealtimeOutputSpeechStartEvent | RealtimeOutputSpeechEndEvent | RealtimeInputTranscriptionErrorEvent | RealtimeSessionReconnectEvent | RealtimeSessionErrorEvent)
What a user may pass as a realtime model’s profile=, mirroring ModelProfileSpec.
Either a partial RealtimeModelProfile merged over the
resolved profile, or a callable taking the resolved profile and returning the one to use, for full
control.
Default: TypeAliasType('RealtimeModelProfileSpec', 'RealtimeModelProfile | Callable[[RealtimeModelProfile], RealtimeModelProfile]')
The names of the RealtimeModelSettings fields, as
ModelSettingName is for ModelSettings.
Default: TypeAliasType('RealtimeModelSettingName', Literal['max_tokens', 'parallel_tool_calls', 'async_tool_calls', 'tool_choice', 'input_transcription_model', 'output_modality', 'output_speech_timestamps', 'thinking', 'turn_detection', 'handshake_timeout', 'reconnect'])
Known values for the OpenAI-protocol models’ input_transcription_model, pinned by a provider sync test.
'auto' is the sentinel that resolves to the provider’s recommended transcription model; the rest are
concrete model ids. The values span providers, so an id valid for one provider (e.g. 'grok-transcribe'
for xAI) is rejected by another at connect time. The field also accepts any other str, so a newer id
not listed here still works — this is just an autocomplete aid, like
KnownModelName.
Default: TypeAliasType('KnownRealtimeTranscriptionModelName', Literal['auto', 'whisper-1', 'gpt-4o-transcribe', 'gpt-4o-mini-transcribe', 'gpt-realtime-whisper', 'gpt-live-transcribe', 'gpt-transcribe', 'grok-transcribe', 'azure-speech', 'mai-transcribe'])
Known realtime model identifiers, surfaced for autocomplete and pinned to provider aliases by a sync test.
Default: TypeAliasType('KnownRealtimeModelName', Literal['openai:gpt-live-1', 'openai:gpt-realtime', 'openai:gpt-realtime-2.1', 'openai:gpt-realtime-2.1-mini', 'azure:gpt-realtime', 'xai:grok-voice-latest', 'xai:grok-voice-think-fast-2.0', 'elevenlabs:eleven_realtime_v1_max', 'elevenlabs:eleven_realtime_v1_mini', 'google:gemini-2.5-flash-native-audio-latest', 'google:gemini-3.1-flash-live-preview', 'google:gemini-3.8-live', 'google:gemini-3.8-live-extended-thinking'])