pydantic_ai.realtime.google
The Gemini Live API provider. Requires the google optional group
(pip install "pydantic-ai-slim[google]").
GoogleRealtimeModel runs over the google-genai
SDK (which manages the WebSocket transport). Gemini expects 16 kHz PCM input (output is 24 kHz),
produces one response modality per session, and natively accepts live video frames sent as
BinaryImage. It exposes Gemini Live’s session and generation
configuration through GoogleRealtimeModelSettings —
shared turn-taking via TurnDetection, with finer Gemini-specific
control via AutomaticVAD in google_vad plus
google_activity_handling/google_turn_coverage, voice via google_voice or a
MultiSpeaker in google_multi_speaker,
and long-session ContextCompression — with
resilience via session resumption + a ReconnectPolicy in the
reconnect setting.
Gemini Live API provider for realtime speech-to-speech (and live video) sessions.
Built on the google-genai SDK, which manages the WebSocket transport for you. Available via the
google optional group:
pip install “pydantic-ai-slim[google-realtime]”
Unlike the OpenAI provider, Gemini wants 16 kHz PCM input audio (output is 24 kHz), produces a
single response modality per session (audio or text), and natively accepts a stream of video
frames sent as BinaryImage.
Use provider='google' for the Gemini Developer API, or provider='google-cloud' /
GoogleCloudProvider for Google Cloud with
Application Default Credentials.
Bases: TypedDict
Server-side voice activity detection — the default turn-taking mode for Gemini Live.
Turn off automatic VAD entirely. Defaults to False.
Do not set this through RealtimeSession: Pydantic AI does not expose Gemini activity markers or
manual turn controls. Use automatic VAD instead; the shared turn_detection=False setting is
rejected for the same reason.
Type: bool
How readily speech onset is detected. high triggers on quieter audio; low is stricter.
Defaults to the provider default.
Type: Literal[‘high’, ‘low’]
How readily the end of speech is detected. high ends turns sooner; low waits longer.
Defaults to the provider default.
Type: Literal[‘high’, ‘low’]
Audio to include before detected speech, in milliseconds. Defaults to the provider default.
Type: int
Silence required to detect the end of speech, in milliseconds. Defaults to the provider default.
Type: int
Bases: TypedDict
Assign prebuilt voices to named speakers for multi-speaker audio output.
Mapping of speaker label to prebuilt voice name, e.g. {'Joe': 'Puck', 'Jane': 'Kore'}.
Defaults to an empty mapping.
Bases: TypedDict
Sliding-window context compression so long sessions don’t exceed the context window.
Compress once the context passes this many tokens. Defaults to the provider default.
Type: int
Target size (in tokens) of the retained sliding window after compression. Defaults to the provider default.
Type: int
Bases: RealtimeModelSettings
Settings used for a Gemini Live session.
Amount of randomness injected into the response.
Type: float
Nucleus sampling probability mass.
Type: float
Only sample from the top K options for each subsequent token.
Type: int
The random seed to use for the session.
Type: int
The thinking configuration to use for the model.
Type: genai_types.ThinkingConfigDict
The video resolution to use for the model.
Type: genai_types.MediaResolution
BCP-47 language code for audio output.
Type: str
Prebuilt voice used for audio output, e.g. Puck.
Type: str
Per-speaker voice assignments; takes precedence over google_voice.
No Gemini Live model supports this: google-genai refuses a multi-speaker voice config on the Live
path outright (ValueError: multi_speaker_voice_config is not supported in the live API), so setting
it raises rather than assigning voices. Multi-speaker output is a
text-to-speech feature; a Live session has one voice, set with google_voice.
Type: MultiSpeaker
Whether to enable emotion-aware delivery.
Not supported by the Gemini 3.1 Flash Live and 3.8 Live models, so connect raises
UserError if it’s enabled for one of them (see
google_supports_affective_dialog).
Type: bool
Whether the model may decide when to respond, including staying silent on input not addressed to it. Useful for “react to the camera” experiences.
Always on for the Gemini 3.8 Live models, so there it can be left unset. They reject an explicit
False, which is never sent: False just leaves the field out.
Gemini serves proactivity on the Developer API’s v1alpha only, and the API version belongs to
the client, so the client has to be built for it — connect raises
UserError naming the fix rather than letting the session fail
to open. Unavailable on Vertex AI, whose version line has no v1alpha.
Type: bool
Whether to transcribe input audio. Defaults to True.
When False, user turns are recorded as retained audio when available, or as content-less
placeholders otherwise. Takes precedence over the shared
input_transcription_model,
whose None also turns transcription off here.
Type: bool
Whether to transcribe output audio. Defaults to True.
When False, retain output audio if assistant audio turns need to appear in history. Assistant
audio without a transcript cannot be handed off or seeded.
Type: bool
Language hints applied to input and output transcription.
Gemini-specific server-side voice activity detection settings.
When present, this fully overrides the cross-provider turn_detection setting.
google_vad={'disabled': True} raises a UserError, like turn_detection=False: Pydantic AI does
not expose Gemini activity markers or manual turn controls, so the resulting session could not
drive turns.
Type: AutomaticVAD
Whether detected user activity interrupts the model.
Type: Literal[‘interrupts’, ‘no_interruption’]
Which realtime input is attached to a turn — 'activity_only', 'all_input' (everything
between turns too), or 'all_video' (all video frames plus audio during activity; ideal for
live-camera use). Absent uses the provider default.
Type: Literal[‘activity_only’, ‘all_input’, ‘all_video’]
Sliding-window context compression for long-running sessions.
Type: ContextCompression
Raw values merged last into the Google LiveConnectConfig.
Whether to request session-resumption handles, which let a re-dial restore the server-side conversation.
When absent, handles are requested exactly when a
reconnect policy is set. An explicit
False cannot be combined with a reconnect policy: a re-dial without resumption would lose the
conversation, so connect raises UserError rather than
silently reconnecting into a model that remembers nothing.
Type: bool
Deprecated: use the shared async_tool_calls setting instead.
Translated (with a deprecation warning) when a session connects; an async_tool_calls in the same
settings wins.
Type: bool
Bases: RealtimeModelProfile
Profile for Gemini Live models, adding the Gemini-specific fields to the shared realtime profile.
Mirrors the GoogleModelProfile /
ModelProfile split on the request-response side.
Thinking levels the Live model accepts. Default: unset.
Same meaning as google_thinking_levels
on a standard model: unset means the full GOOGLE_THINKING_LEVELS
scale, and a unified thinking effort snaps to the
nearest level in the set.
Type: frozenset[GoogleThinkingLevel]
Whether the model always reasons, so its API requires a thinking level. Default: False.
Mirrors ModelProfile.thinking_always_enabled.
A session that sets no thinking still sends
the cheapest level the model accepts, and thinking=False means “as little as possible” rather than a
thinking_budget=0 the model would reject. gemini-3.8-live-extended-thinking closes the handshake
with 1007 Thinking level must be specified for this model without a level.
Type: bool
Whether the model runs a tool call asynchronously when its declaration sets no behavior. Default: False.
True of the Gemini 3.8 Live family, where Google made NON_BLOCKING the default. Tool calls stay
blocking unless async_tool_calls
asks otherwise, so on such a model the declaration says BLOCKING explicitly instead of leaving it unset.
Type: bool
Deprecated: use async_tool_call_mode='always' instead.
Translated (with a deprecation warning) when the profile is resolved: True becomes
async_tool_call_mode='always', and False, which left the choice to the other flags, is dropped.
Type: bool
Whether the model closes a tool-call turn with a turn_complete of its own. Default: False.
Vertex’s half-cascade gemini-live-2.5-flash sends one when the tool-call generation ends (usage
only, no output), whether or not the results have arrived yet, and another after speaking the
answer (verified live); other Live models send only the answer’s. With
this set, the first of the two is reported as the tool-call response’s usage rather than a turn
boundary, so the exchange isn’t reported complete before the answer is spoken. Set by default on
Vertex AI only, where it was verified.
Type: bool
Whether the model takes a scheduling field on an async tool call’s result. Default: False.
Separate from whether the call runs asynchronously at all: that is decided when the call is
declared, while scheduling says how its result enters the speech the model is producing when it
arrives. Pydantic AI sends FunctionResponseScheduling.INTERRUPT, so the result cuts in rather
than waiting for the model to go idle. gemini-3.8-live-extended-thinking runs every call
asynchronously but paces results against its own reasoning, and closes the session with 1007 Function response scheduling is not supported for this model if the field is sent at all.
Type: bool
Whether the model takes google_affective_dialog. Default: True.
When False, connect raises UserError for a session that
enables it, rather than opening one the provider rejects. False for the gemini-3.1-flash-live and
gemini-3.8-live families, which don’t support affective dialog: gemini-3.1-flash-live-preview
refuses the handshake with 1007 Request contains an invalid argument, and the 3.8 models open the
session and then close it with the same error on the first send.
Type: bool
Media types a tool result can carry inside its function response. Default: ().
The realtime counterpart of
google_supported_mime_types_in_tool_returns
on a standard model: content of these types attached to a tool return (a
BinaryContent or a downloaded
ImageUrl) goes in FunctionResponse.parts, and any other media
raises UserError with the result unsent. PNG, JPEG, WebP, and
plain text on the Gemini 3.x Live models, which read them; gemini-2.5-flash-native-audio-*
doesn’t, and the 3.x models close the session on a PDF.
Whether seeded tool calls and results go in as native function parts. Default: False.
When True, prior ToolCallParts,
ToolReturnParts, and tool
RetryPromptParts are seeded as function_call and
function_response parts, sent as the session’s initial history (history_config), rather than
projected as readable text. True of the Gemini 3.8 Live models. gemini-2.5-flash-native-audio-*
rejects function parts in seeded turns, and gemini-3.1-flash-live-preview loses history seeded
that way when a reconnect resumes the
session.
Type: bool
Whether a typed turn sees an image sent just before it as a live video frame. Default: True.
session.send(image) sends the image as a video frame, which a spoken turn sees. On a model where
a typed turn doesn’t, the most recent image sent in the last 10 seconds is sent again in the typed
turn’s own content, ahead of the text. Every Gemini Live model probed misses it:
gemini-3.1-flash-live-preview, gemini-3.8-live and its extended-thinking variant answer that
they can’t see an image, and gemini-2.5-flash-native-audio-* and Vertex’s gemini-live-2.5-flash
misread it.
Type: bool
Bases: RealtimeModel
Gemini Live API model.
Session and generation configuration is read from
GoogleRealtimeModelSettings, passed
through settings as model-level defaults or as model_settings when opening a session.
Authentication and the underlying google-genai client come from a
Provider, mirroring GoogleModel.
Pass provider='google' (the default) for the Gemini Developer API (reads GOOGLE_API_KEY /
GEMINI_API_KEY), provider='google-cloud' for Vertex AI (Application Default Credentials, useful
where org policy disallows API keys), or a GoogleProvider /
GoogleCloudProvider instance for a custom
key, client, or region. Gemini Live is available on both surfaces.
The model name, e.g. gemini-3.8-live (low-latency voice), gemini-3.8-live-extended-thinking
(reasons in the background while it speaks, and always runs tools asynchronously), or
gemini-2.5-flash-native-audio-latest (an alias that tracks the newest native-audio Live model).
provider : Literal[‘google’, ‘google-cloud’, ‘gateway’] | Provider[Client] Default: 'google'
The provider to use for authentication and API access — 'google' (Gemini Developer
API, the default) or 'google-cloud' (Vertex AI), or a Provider instance.
settings : RealtimeModelSettings | None Default: None
Model-level defaults for session and generation configuration.
profile : RealtimeModelProfileSpec | None Default: None
Optional override for the realtime model profile,
merged over the provider’s — a partial dict, or a callable taking the resolved profile and
returning the one to use. Mirrors profile= on a standard
Model, and is the escape hatch when a model name doesn’t
identify the model (e.g. an Azure deployment named something other than its model).
The underlying google.genai.Client from the provider.
Type: Client
Bases: RealtimeConnection
A live connection to the Gemini Live API, backed by a google-genai session.
@async
def send(content: RealtimeInput) -> None
Send content to the Gemini Live API.
Accepts BinaryAudio (raw PCM16, 16kHz, mono), a str text turn, TextContext (text sent
with turn_complete=False, so it waits for the next turn), BinaryImage (a live video
frame), and ToolResult. The manual turn-taking verbs are not supported (Gemini uses
automatic VAD).
Sample rate (Hz) Gemini expects for PCM16 input audio.
Default: 16000