codec
The lower-level codec vocabulary, for implementing a realtime provider or consuming a
RealtimeConnection directly: the raw events a
connection yields, the turn-control verbs and inputs it accepts, and the model-profile merge helpers.
Most users only need the session-level API in pydantic_ai.realtime.
Low-level codec vocabulary for realtime providers.
Most users only need the session-level API in pydantic_ai.realtime
(AgentRealtime.session, the events a session yields,
and the content passed to RealtimeSession.send). This
submodule holds the lower-level vocabulary used when implementing a realtime provider or consuming a
RealtimeConnection directly: the raw codec events a
connection yields to the session, the turn-control verbs and inputs a connection accepts, and the
model-profile merge helpers.
The result of a tool call, rendered for the wire and sent back to the model.
Built and sent by RealtimeSession after it settles a
call: the string-only realtime tool channel and the retry/failure error-key wrapping mean the
session renders the ToolReturnPart or
RetryPromptPart it records in history down to this flat
shape, so every provider sends exactly the same rendering.
Identifier of the ToolCall this result answers.
Type: str
The tool’s output, rendered as a string.
Type: str
Additional user content to send after the tool output when the provider supports it.
Type: Sequence[UserContent] | None Default: None
Commit the buffered input audio as a user turn (manual turn-taking / push-to-talk).
Only needed when automatic voice activity detection is disabled; with server-side VAD the provider commits audio and triggers a response automatically.
Discard any buffered, uncommitted input audio.
Ask the model to generate a response now (manual turn-taking, after CommitAudio).
Cancel the model’s in-progress response (maps to the provider’s response-cancel).
Truncate the model’s current audio output at audio_end_ms.
After a barge-in the user only heard part of the model’s audio. Truncating tells the provider how much was actually played, so its stored transcript matches and the conversation context stays consistent. The provider resolves which output item to truncate from its own state.
Milliseconds of the current output audio that were actually played before the interruption.
Type: int
A chunk of audio output from the model.
Raw PCM audio bytes. The sample rate is provider-specific.
Type: bytes
Provider item ID for the spoken output this chunk belongs to, when available.
Type: str | None Default: None
The model’s textual output (partial or final): an audio transcript, or plain text output.
Transcript text. A partial event carries the incremental delta; a final event the full turn.
Type: str
Whether this is the final transcript for the turn.
Type: bool Default: False
Whether this is the model’s plain text output (output_modalities=('text',)) rather than a
transcription of spoken audio. Text output becomes a TextPart;
an audio transcript becomes a SpeechPart.
Type: bool Default: False
Provider item ID for the spoken output, when available.
Type: str | None Default: None
A transcription of the user’s audio input (partial or final).
Providers with per-item IDs use item_id to associate interleaved transcripts with the correct
user turn. Providers without them retain arrival-order association.
Transcript text.
Type: str
Whether this is the final transcript for the user’s turn.
Type: bool Default: False
Provider item ID for the user’s turn, when available.
Type: str | None Default: None
Whether text is the whole transcript so far rather than an incremental piece.
Speech recognition is revisable, and a provider that streams cumulative snapshots may correct
what it already transcribed instead of only extending it. Setting this lets the session adopt
each snapshot as authoritative rather than guessing from prefixes whether the text appends;
it surfaces the difference to callers as a
SpeechPartDelta.transcript carrying the
corrected whole. Leave False for incremental deltas.
Type: bool Default: False
The model is requesting a tool call.
Provider-assigned identifier for this call.
Type: str
Name of the tool to invoke.
Type: str
Raw JSON-encoded arguments. May be an empty string if the model sent no arguments.
Type: str
Whether per-response SessionUsage will follow
this call before the provider’s response is complete.
OpenAI-protocol providers report calls before response.done, which carries usage; the session
uses this signal to keep all calls and their usage on the same ModelResponse.
Type: bool Default: False
Provider conversation-item ID for this call, when available.
Type: str | None Default: None
The model cancelled in-flight tool calls (e.g. the user barged in before they finished).
Gemini Live sends this as toolCallCancellation; the session cancels the matching running tool
tasks so their now-unwanted results are never sent back to the model.
Identifiers of the ToolCalls that were cancelled.
The provider reported that its current response is done.
This codec event is consumed by the session to finalize a
ModelResponse. Providers do not necessarily report a
terminal for every model response the session records in history.
Whether the response ended because it was cancelled (e.g. the user barged in).
Type: bool Default: False
Provider-assigned ID for the completed response, when available.
Type: str | None Default: None
Normalized reason the provider finished the response, when available.
Type: FinishReason | None Default: None
Raw provider terminal status details retained on the finalized response, when available.
Type: dict[str, Any] | None Default: None
Event type identifier, used as a discriminator.
Type: Literal[‘response_done’] Default: 'response_done'
Usage reported by the provider for a model response or another run-level operation.
Normalized usage ready to accumulate into a RunUsage.
Type: RequestUsage
Provider-assigned ID for the response this usage belongs to, when available.
Type: str | None Default: None
Normalized completion reason for the response this usage belongs to, when available.
Type: FinishReason | None Default: None
Whether this usage belongs to a specific model response.
True, the default, accumulates it both into the run total and the response’s
ModelResponse.usage. False is run-level only, e.g. input audio transcription usage,
which is billed on a separate model/meter and is accumulated into the run’s RunUsage
but attributed to no ModelResponse.
Type: bool Default: True
Event type identifier, used as a discriminator.
Type: Literal[‘session_usage’] Default: 'session_usage'
An OpenAI-protocol server assigned a conversation ID.
This is a codec-level control event. Providers consume it during their handshake when possible; the session silently consumes any instance that reaches the live stream.
Provider-assigned conversation ID.
Type: str
An OpenAI-protocol server reported a conversation item.
xAI uses replayed=True for item events emitted during the resume handshake. The session consumes
those items and remembers their newly assigned IDs so any follow-on content or tool events aren’t
appended or executed again.
Provider-assigned conversation-item ID, when present.
Type: str | None Default: None
Provider-assigned tool-call ID, for function call and result items.
Type: str | None Default: None
Whether the provider identified this item as part of a resumption replay.
Type: bool Default: False
Bases: ABC
A live connection to a realtime model.
Providers implement this to handle protocol-specific framing (WebSocket frames,
HTTP/2 messages, etc.). Content is fed in via send
and events are consumed by iterating the connection.
The exception types this connection’s transport raises when the link to the provider fails.
A RealtimeSession maps these to
RealtimeError so a failed send surfaces as the same typed
error as a failed receive, instead of leaking a websockets or provider-SDK exception the caller
has no reason to expect from a model call. Leave empty if send already raises typed errors.
The mapping covers the whole of send, so
anything it does besides writing to the transport — converting content, downloading media —
must not raise these types, or a local failure would be reported as a lost connection. Do that
work before the first frame goes out.
Type: tuple[type[Exception], …] Default: ()
The model id the server reported serving this session, when the provider reports one.
Captured from the connect handshake (e.g. the OpenAI protocol’s session.created). It can
differ from the requested model id: xAI accepts any model slug and silently substitutes its
current default, reporting the actually-served model only here. None when the provider
doesn’t report one (e.g. Gemini Live). The session stamps this on each
ModelResponse.model_name, mirroring how
request-response models record the response’s reported model rather than the requested one.
Whether this connection will emit InputTranscript events for the user’s audio.
Providers that transcribe the user’s input (the default) leave this True. When it is False,
no transcript arrives, so RealtimeSession finalizes a
user turn from retained input audio instead (see audio_retention). Defaults to True so a
connection that doesn’t override it never triggers the audio-only path (which would risk a
duplicate turn if transcripts did arrive).
Type: bool
Whether a reconnect continues the response and tool calls that were in flight when the socket dropped.
Otherwise it only brings back the finalized conversation. Native session resumption (xAI Grok Voice) restores the in-flight generation server-side, and
Gemini Live settles the cut turn in the connection before its
RealtimeSessionReconnectEvent, so in
both the RealtimeSession must not settle again and
trusts state_restored. Local replay (OpenAI, Azure OpenAI) restores only finalized turns, so
the session settles the interrupted turn itself and reports state_restored=False. Defaults to
True; the OpenAI connection overrides it.
Type: bool
@abstractmethod
@async
def send(content: RealtimeInput) -> None
Feed content into the session.
Concrete connections accept provider-specific data and control inputs. OpenAI accepts audio,
text, images, tool results, manual turn controls, cancellation, and truncation; Gemini accepts
audio, text, images, and tool results. A high-level
RealtimeSession checks profile-gated operations and
raises UserError, as does a connection handed an input it
can’t send.
@abstractmethod
def __aiter__() -> AsyncIterator[RealtimeCodecEvent]
Iterate over events received from the model.
AsyncIterator[RealtimeCodecEvent]
def set_message_history(message_history: Callable[[], Sequence[ModelMessage]]) -> None
Tell the connection how to read the message history as it currently stands.
A RealtimeSession calls this when it takes ownership of
the connection, so a provider that loses server-side state on
reconnect can replay the conversation into the new
session instead of resuming with total amnesia. The session’s history grows as the call goes on,
hence a callable rather than a snapshot.
A no-op by default: providers with native session resumption (Gemini Live, xAI) have nothing to replay, and one that can’t seed a session at all has nowhere to put it.
def merge_realtime_profile(
base: RealtimeModelProfile | None,
*overrides: RealtimeModelProfile | None,
) -> RealtimeModelProfile
Merge realtime profiles, with later layers overriding earlier ones.
RealtimeModelProfile
The content types a caller feeds into RealtimeSession.send.
Session content only, in the shared message vocabulary: a str is a complete text turn, and
BinaryContent carries an image frame, WAV audio (unwrapped to
raw PCM before it is streamed, matching the history-seeding path), or a raw PCM chunk
(media_type='audio/pcm'). The session normalizes these before forwarding them to the connection.
Turn-control verbs (CommitAudio, ClearAudio, CreateResponse, CancelResponse,
TruncateOutput) are connection-level vocabulary driven through the dedicated RealtimeSession
methods (commit_audio(), clear_audio(), create_response(), interrupt()), and
ToolResult is sent by the session itself when a tool
completes — neither is accepted by send().
Default: TypeAliasType('RealtimeSessionInput', 'str | BinaryContent')
The sample rate, in Hz, assumed for PCM audio when a realtime model profile doesn’t specify one.
Default: 24000
Default realtime model profile values.
Type: RealtimeModelProfile Default: {'supports_image_input': False, 'supports_manual_turn_control': False, 'supports_interruption': False, 'supports_output_truncation': False, 'supports_text_output': True, 'supports_session_seeding': False, 'supports_webrtc': False, 'supports_seeding_images': False, 'supports_seeding_audio': False, 'supports_async_tool_calls': False, 'supports_tool_return_schema': False, 'supported_native_tools': frozenset(), 'emits_input_speech_events': False, 'audio_input_sample_rate': DEFAULT_AUDIO_SAMPLE_RATE, 'audio_output_sample_rate': DEFAULT_AUDIO_SAMPLE_RATE}
Union of content types accepted by RealtimeConnection.send.
The connection-level counterpart of RealtimeSessionInput,
already normalized: a str is a complete text turn, a
BinaryAudio carries a raw mono PCM16 chunk at the model’s
audio_input_sample_rate
(media_type='audio/pcm'), and a BinaryImage an image frame.
The connection additionally accepts the turn-control verbs and
ToolResult, which
RealtimeSession sends on the caller’s behalf.
Default: TypeAliasType('RealtimeInput', 'str | BinaryAudio | BinaryImage | CommitAudio | ClearAudio | CreateResponse | CancelResponse | TruncateOutput | ToolResult')
Union of the low-level codec events yielded by RealtimeConnection.
This is the provider-facing vocabulary: providers translate their wire protocol into these events, and
RealtimeSession translates them again into the shared
RealtimeEvent vocabulary while building
ModelMessage history.
Default: TypeAliasType('RealtimeCodecEvent', AudioDelta | OutputTranscript | InputTranscript | ToolCall | ToolCallCancelled | ResponseDone | RealtimeInputSpeechStartEvent | RealtimeResponseInterruptedEvent | RealtimeInputSpeechEndEvent | RealtimeOutputSpeechStartEvent | RealtimeOutputSpeechEndEvent | RealtimeInputTranscriptionErrorEvent | SessionUsage | RealtimeSessionReconnectEvent | ConversationCreated | ConversationItemCreated | PartStartEvent | PartEndEvent | RealtimeSessionErrorEvent)