Google Gemini Live

GoogleRealtimeModel connects an agent to Gemini Live, including native audio, live images, and provider-native tools. Start with the realtime quickstart or camera example.

Setup

To use Gemini Live models, install pydantic-ai-slim with the google-realtime optional group, which bundles the google-genai SDK together with the realtime transport dependencies:

pip

Terminal

pip install "pydantic-ai-slim[google-realtime]"

uv

Terminal

uv add "pydantic-ai-slim[google-realtime]"

Authentication comes from provider, mirroring GoogleModel. Use provider='google' for the Gemini Developer API or provider='google-cloud' for Vertex AI/ADC, with API keys and credentials configured as described in the Google model documentation. Pass a GoogleProvider or GoogleCloudProvider for custom credentials, project, region, or client.

Model names

Use a Gemini Live model ID, for example gemini-3.8-live, gemini-3.8-live-extended-thinking, or gemini-2.5-flash-native-audio-latest. Live models differ in thinking, asynchronous tools, and output behavior; the profile resolved from the model ID describes what each one supports. Use the official Gemini Live documentation as the canonical model and availability source.

Vertex AI and the Pydantic AI Gateway use different model IDs and locations from the Gemini Developer API:

API	Model ID	Location
Gemini Developer API	gemini-2.5-flash-native-audio-latest	n/a (no location)
Gemini Developer API	gemini-3.1-flash-live-preview	n/a (no location)
Gemini Developer API	gemini-3.8-live	n/a (no location)
Gemini Developer API	gemini-3.8-live-extended-thinking	n/a (no location)
Vertex AI / gateway	gemini-live-2.5-flash	global
Vertex AI / gateway	gemini-live-2.5-flash-native-audio	us-central1

The Developer API IDs are not available on Vertex AI. Configure the matching Vertex location on GoogleCloudProvider or in the gateway; the gateway example below uses gemini-live-2.5-flash and therefore requires global.

Extended thinking

gemini-3.8-live-extended-thinking reasons in the background while it keeps talking, so it can speak a filler (“Let me check those flights for you”), run a tool, and answer — all within one exchange. It differs from every other Live model in three ways the model handles for you:

• It requires a thinking level. One is sent even when the session asks for nothing, at the cheapest level the model accepts, because reasoning costs latency. Ask for more with thinking='medium' or 'high'. thinking=False means “as little as possible” here rather than “off”, since the model has no off.
• Its tool calls are always asynchronous. Its profile’s async_tool_call_mode is 'always', since the model has no blocking mode, so an explicit async_tool_calls=False is ignored, with an UnsupportedSettingWarning when the session has tools.
• Its spoken filler doesn’t end the turn. Gemini closes the filler’s response and says the interaction is still in progress, so the filler and the tool call it was stalling for are recorded as one ModelResponse, and RealtimeTurnCompleteEvent and wait_for_reply() both wait for the model to actually finish. A UI that shows “thinking…” should key off those. An utterance the user barges in on is the exception: it really is over, and is recorded as its own interrupted response.

gemini-3.8-live is the same family without background reasoning. It rejects a thinking level, so the shared thinking setting is ignored rather than sent. Unlike older Live models it defaults to asynchronous tool calls, so Pydantic AI declares tools BLOCKING unless async_tool_calls=True asks otherwise, keeping the same default as every other model.

Both 3.8 models keep proactive audio permanently on, so there’s nothing for google_proactive_audio to turn on and it can be left unset. Gemini rejects an explicit False, which Pydantic AI never sends, and True is accepted too. Neither supports affective dialog, and neither does gemini-3.1-flash-live-preview: google_affective_dialog=True raises UserError when the session connects.

Settings

GoogleRealtimeModelSettings — the realtime counterpart of model run settings — extends the shared settings with Google generation and Live controls:

from pydantic_ai.realtime.google import GoogleRealtimeModel, GoogleRealtimeModelSettings

settings = GoogleRealtimeModelSettings(
    temperature=0.7,
    top_p=0.9,
    google_voice='Puck',
    google_language_code='en-US',
    google_affective_dialog=True,
    google_vad={'start_sensitivity': 'high', 'end_sensitivity': 'low'},
    google_turn_coverage='all_video',
    google_context_compression={'trigger_tokens': 16000, 'target_tokens': 8000},
)
model = GoogleRealtimeModel('gemini-2.5-flash-native-audio-latest', settings=settings)

Setting	Purpose
google_voice, google_language_code	Voice and output language. A Live session speaks in one voice: the deprecated google_multi_speaker raises when the session connects
google_affective_dialog	Emotion-aware delivery, on the 2.5 models (not 3.1 Flash Live or the 3.8 models)
google_proactive_audio	Model-decided speech on native-audio models, served on the Developer API’s v1alpha (see below). Always on for the 3.8 models
google_vad	Exact automatic VAD; fully overrides shared turn_detection
google_activity_handling, google_turn_coverage	Interruption behavior and which input belongs to a turn
google_input_transcription, google_output_transcription	Native transcription switches, enabled by default
google_context_compression	Sliding-window compression for long sessions
google_enable_session_resumption	Native state restoration; enabled automatically by a reconnect policy
google_config_overrides	Raw LiveConnectConfig keys merged last as a forward-compatibility escape hatch

google_voice is the provider voice setting. google_thinking_config takes precedence over the shared thinking setting when a token budget or other Gemini-specific control is needed.

Gemini Live has no tool-choice configuration, so it can’t force a tool call: a tool_choice of 'required' or a list of tool names raises a UserError before connecting. To limit the model to some tools, pass [ToolOrOutput(function_tools=[...])][pydantic_ai.settings.ToolOrOutput] instead. It also can’t stop the model from calling several tools at once, so parallel_tool_calls=False warns with an UnsupportedSettingWarning.

google_proactive_audio dials the v1alpha API

Gemini serves proactivity on the Developer API’s v1alpha only; on any other version the session is closed with 1007 Invalid JSON payload received. Unknown name "proactivity" at 'setup'. When the provider was built from an API key (the default 'google' provider, or GoogleProvider(api_key=...)), a session with google_proactive_audio=True opens its Live connection with a separate v1alpha client built from that key, closed when the session ends. The provider’s own client is left as it is, so ordinary GoogleModel requests sharing it keep their version.

A client you built yourself can’t be rebuilt that way, so pass one that is already on v1alpha; connecting with one on another version raises UserError:

proactive_audio.py

from google.genai import Client, types

from pydantic_ai.providers.google import GoogleProvider
from pydantic_ai.realtime.google import GoogleRealtimeModel, GoogleRealtimeModelSettings

client = Client(
    api_key='your-api-key', http_options=types.HttpOptions(api_version='v1alpha')
)
model = GoogleRealtimeModel(
    'gemini-2.5-flash-native-audio-latest',
    provider=GoogleProvider(client=client),
    settings=GoogleRealtimeModelSettings(google_proactive_audio=True),
)

Unavailable on Vertex AI, whose version line has no v1alpha.

Keep automatic VAD enabled

Pydantic AI does not expose Gemini activity markers or manual turn verbs. Do not set google_vad={'disabled': True}; shared turn_detection=False is rejected for the same reason.

Asynchronous tool calls

Gemini normally pauses generation while a function tool is outstanding. On the native-audio models and gemini-3.8-live, set the shared async_tool_calls setting to True to let it keep speaking and answering: tools are then declared NON_BLOCKING, and their results are returned with INTERRUPT scheduling, so each cuts into whatever the model is saying when it arrives. This is best for slow tools, since a fast result can cut off speech that barely started. What the model says while the tool runs is recorded with the call, ahead of its result: see Transcription and history edge cases.

The other Live models accept a NON_BLOCKING declaration and then wait for the result anyway, so they ignore the setting, and gemini-3.8-live-extended-thinking, which has no blocking mode, runs every tool call this way regardless — see Extended thinking. The deprecated google_async_tool_calls setting still works as an alias, with a deprecation warning.

Native tools

Gemini Live maps WebSearch to Google Search grounding, the only native tool it supports — no Live model runs native code execution or URL context, so neither CodeExecutionTool nor WebFetch is advertised in supported_native_tools. Give those a local= fallback and the session runs the local tool instead: NativeOrLocalTool(native=CodeExecutionTool(), local=...), or WebFetch(native=False, local=True), which requires the web-fetch optional group (pip/uv-add "pydantic-ai-slim[google-realtime,web-fetch]"). Google Search grounding takes no domain filters or usage limit, so a WebSearch with allowed_domains, blocked_domains, max_uses, or external_web_access=False set fails before the session connects (see restricting parameters).

Specialist streaming models

Gemini also serves specialist streaming models on the same endpoint. The robotics ones, like gemini-robotics-er-2-streaming-preview, are perception models rather than voice assistants: they take in a live audio and video stream and answer in text, for a robot’s controller to act on, and reject audio output. Their profile reports supports_audio_output=False, so a session on one asks for text by default and records the answers as TextParts:

robot_perception.py

from pydantic_ai import Agent

perception = Agent(instructions='You are the perception module of a robot.')


async def main(frames=()):
    async with perception.realtime('google:gemini-robotics-er-2-streaming-preview').session() as session:
        for frame in frames:
            await session.send(frame)  # a `BinaryImage` per camera frame
        await session.send('What object is in front of you?')
        await session.wait_for_reply()

For a streaming model the built-in profile doesn’t describe, correct the facts with profile=, which resolves like a standard model profile.

Feature support and limitations

Feature	Support	Notes
Audio format	Full feature support	Mono PCM16, 16 kHz input and 24 kHz output
Text output	Vertex gemini-live-2.5-flash and the robotics streaming models only	The other Live models reject a TEXT response modality, so output_modality='text' raises; read the answer from the transcript on the SpeechPart. The robotics models only write text, which is their default
Image/live video input	Full feature support	Images; google_turn_coverage='all_video' keeps streamed frames in context
Manual turns	Unsupported	Automatic turn detection is required
Explicit interruption/truncation	Unsupported	Gemini interrupts server-side and emits RealtimeResponseInterruptedEvent
Input transcription	Full feature support	Native transcription, enabled by default; no separate model ID
Native tools	Limited parameter support	Google Search grounding only; URL context and code execution fall back to a local= tool (see above)
Usage	Full feature support	Token and modality breakdowns; function-call usage may arrive on a later turn
State-restoring reconnect	Full feature support	Requires session resumption plus a reconnect policy

See Audio, images, and transcripts, Turns and interruptions, Tools, and Connection lifecycle for the provider-agnostic workflows.

Gateway

To route through the Pydantic AI Gateway, use a gateway/-prefixed model string:

from pydantic_ai import Agent

agent = Agent(instructions='You are a helpful voice assistant.')
realtime = agent.realtime('gateway/google:gemini-live-2.5-flash')

The gateway proxies Gemini Live through the Vertex upstream, so configure a region that supports the selected model as listed in Model names. gateway/google-cloud is an alias. See Gateway trace propagation.

Session resumption

For state-restoring reconnects, set the reconnect setting to a ReconnectPolicy; session resumption is enabled automatically alongside it (google_enable_session_resumption can still request handles without a policy, and explicitly setting it to False next to a policy raises UserError rather than silently losing the conversation). Reconnection uses the latest in-memory server handle and emits state_restored=True, unless the drop cut off an exchange the resumed session no longer has (see State restoration). A drop before the server has issued a handle dials a fresh session instead and replays the session’s local history into it, as on the other providers.

The connection cap can briefly interrupt a turn

Gemini sends GoAway shortly before its provider-defined connection cap. Pydantic AI reconnects after the drop rather than proactively on GoAway, so a long call can briefly drop mid-turn. This is tracked in #6643.

Provider-specific quirks

• What happens to text (or a tool result) that reaches the model while it generates a reply depends on the model:

  • The native-audio models finish the reply, and answer the text in a turn after it.
  • gemini-3.1-flash-live-preview, gemini-3.8-live, and Vertex’s gemini-live-2.5-flash cut the reply off: they emit RealtimeResponseInterruptedEvent, the partial reply is recorded as interrupted, and the next turn answers the text. gemini-3.1-flash-live-preview also cancels the tool calls the cut-off turn was waiting on, and makes them again.

  wait_for_reply() and RealtimeTurnCompleteEvent wait for the reply to the text. Where the text went out just as a reply began, before the session saw it start, telling which reply answers the text takes the google_input_cuts_off_reply profile flag: the cut-off reply then answers only what started it. It is on for gemini-3.1-flash-live-preview and gemini-3.8-live, which report where in the audio the user started speaking, telling a reply the user spoke over apart from one text cut off. It is off for Vertex’s gemini-live-2.5-flash, which reports no speech, and for the native-audio models, so there a wait for such text can end with the reply already under way. For a model not covered (gemini-3.8-live-extended-thinking, say) that cuts its reply off this way and reports speech, turn the flag on with profile=.

• Gemini reports response interruption but not user speech-start/end events, so local playback is flushed on RealtimeResponseInterruptedEvent, and Gemini sessions record no user speech span (see Logfire instrumentation).

• When Gemini says why it ended a turn, the response records it the way a GoogleModel response does: a malformed function call is finish_reason='error', refused input or unsafe output is 'content_filter', and the raw reason (such as MALFORMED_FUNCTION_CALL) is kept in provider_details['finish_reason'].

• send() sends an image as a live video frame. Spoken turns see video frames, but typed turns don’t on the Live models. So a typed turn (send('...')) also carries the most recent image sent in the last 10 seconds in its own content, ahead of the text. That image is sent, and counted as input, twice. Context text (respond=False) and audio don’t carry it. To ask about an image in writing, send the two together so the question is always inside the window: session.send([image, 'What is this?']). The google_text_turns_see_video_frames profile flag controls the second send.

• Gemini 3.x Live models transcribe the user’s speech even with input transcription turned off. Pydantic AI discards those transcripts, so the setting still keeps the user’s words out of history, but they are still produced on Google’s side.

• After resuming from a handle it issued at the start of a turn, gemini-3.8-live can stream that turn’s answer without ever sending turn_complete, then answer the next input with RealtimeResponseInterruptedEvent and nothing more. The reply stays open, so wait_for_reply() doesn’t return. This happens on Google’s side, and no client-side workaround is known.

• Native transcription can produce only a completed sentence on some models. Caption UIs should replace text from TranscriptUpdate.transcript rather than assume incremental deltas.