openai
The OpenAI Realtime API provider. Requires the realtime and openai optional groups
(pip install "pydantic-ai-slim[realtime,openai]").
OpenAIRealtimeModelSettings configures
the session, including shared turn-taking via TurnDetection
(or False for push-to-talk). For finer control, openai_turn_detection accepts
ServerVAD or
SemanticVAD and fully overrides the shared setting.
Resilience comes from the reconnect setting: a
ReconnectPolicy in
RealtimeModelSettings.
OpenAI Realtime API provider for speech-to-speech sessions.
Connects to wss://api.openai.com/v1/realtime over a WebSocket and maps the OpenAI event
protocol to the shared realtime event types.
Requires the websockets and openai packages, available via the realtime and openai optional
groups:
pip install “pydantic-ai-slim[openai-realtime]“
Bases: RealtimeModelSettings
Settings specific to OpenAI realtime models.
Voice used for audio output, e.g. alloy or VoiceID(id='voice_1234').
The known prebuilt names provide autocomplete, while any string and the OpenAI SDK’s custom
VoiceID form (openai.types.realtime.realtime_audio_config_output.VoiceID) are also accepted.
Type: KnownOpenAIRealtimeVoiceName | str | VoiceID
Noise reduction tuned for near_field (headset) or far_field (laptop/conference) microphones.
Absent disables it.
Type: Literal[‘near_field’, ‘far_field’]
Playback speed multiplier for generated audio (0.25-1.5).
Type: float
OpenAI-specific server or semantic VAD configuration.
When present, this fully overrides the cross-provider turn_detection setting.
Type: ServerVAD | SemanticVAD
How the session truncates conversation context once it exceeds the model’s window.
'auto' (the server default) drops the oldest turns; 'disabled' keeps everything (and errors
when the window is full); a retention_ratio truncation (\{'type': 'retention_ratio', 'retention_ratio': 0.8\}) keeps a fixed fraction, holding the prompt-cached prefix stable across
turns (cached audio is far cheaper). This is the OpenAI SDK’s truncation shape, forwarded as-is.
Type: RealtimeTruncationParam
Bases: TypedDict
Server-side voice activity detection — the default turn-taking mode.
The server detects when the user starts and stops speaking and (by default) commits the audio and triggers a response automatically. Unset fields fall back to the provider defaults.
The turn-detection type. Must be 'server_vad'.
Type: Required[Literal[‘server_vad’]]
Activation threshold (0.0-1.0). Higher requires louder audio; better in noisy environments. Defaults to the provider default.
Type: float
Audio to include before detected speech, in milliseconds. Defaults to the provider default.
Type: int
Silence required to detect the end of speech, in milliseconds. Defaults to the provider default.
Type: int
Whether to automatically generate a response when the user stops speaking. Defaults to True.
Type: bool
Whether to interrupt an in-progress response when the user starts speaking. Defaults to True.
Type: bool
If set, auto-trigger a response after this much idle time with no detected speech. Defaults to the provider default.
Type: int
Bases: TypedDict
Model-based semantic turn detection — uses a model to decide when the user is done speaking.
The turn-detection type. Must be 'semantic_vad'.
Type: Required[Literal[‘semantic_vad’]]
How eagerly the model responds. Defaults to 'auto'; low waits longer and high responds sooner.
Type: Literal[‘low’, ‘medium’, ‘high’, ‘auto’]
Whether to automatically generate a response when a turn ends. Defaults to True.
Type: bool
Whether to interrupt an in-progress response when the user starts speaking. Defaults to True.
Type: bool
Bases: RealtimeConnection
A live WebSocket connection to the OpenAI Realtime API.
The call so far, when a session has offered it for replay on reconnect.
Type: Callable[[], Sequence[ModelMessage]] | None
@async
def send(content: RealtimeInput) -> None
Send content to the OpenAI Realtime API.
Accepts BinaryAudio (raw PCM16, 24kHz, mono), a str text turn, BinaryImage,
ToolResult, and the control verbs CommitAudio, ClearAudio, CreateResponse,
CancelResponse, and TruncateOutput.
Bases: RealtimeModel
OpenAI Realtime API model.
Authentication and the base URL come from a
Provider, mirroring OpenAIChatModel.
Pass provider='openai' (the default) to read OPENAI_API_KEY / OPENAI_BASE_URL from the
environment, or an OpenAIProvider instance for a
custom key or base URL. The realtime transport is opened separately with websockets, so the
provider’s httpx client is not used for the WebSocket connection. The realtime WebSocket URL is
derived from the provider’s base URL (e.g. https://api.openai.com/v1/ →
wss://api.openai.com/v1/realtime), so OpenAI-compatible endpoints that expose a realtime API
work too.
The model name, e.g. gpt-realtime or gpt-realtime-2.1-mini.
provider : Provider[AsyncOpenAI] | str Default: 'openai'
The provider to use for authentication and the base URL. Defaults to 'openai'.
Azure OpenAI is not supported (its realtime endpoint uses a different URL and auth scheme).
settings : RealtimeModelSettings | None Default: None
Model settings used as defaults for realtime sessions.
profile : RealtimeModelProfileSpec | None Default: None
Optional override for the realtime model profile,
merged over the provider’s — a partial dict, or a callable taking the resolved profile and
returning the one to use. Mirrors profile= on a standard
Model, and is the escape hatch when a model name doesn’t
identify the model (e.g. an Azure deployment named something other than its model).
The underlying AsyncOpenAI client from the provider.
Type: AsyncOpenAI
def map_event(data: dict[str, Any]) -> RealtimeCodecEvent | None
Map a raw OpenAI Realtime event to a RealtimeCodecEvent.
Returns None for events that carry no session-relevant content (e.g. session.created).
RealtimeCodecEvent | None
The prebuilt voices OpenAI’s realtime API ships, mirroring the openai SDK’s own Voice union.
The openai_voice setting also
accepts any other string, so a voice OpenAI adds later works before this list catches up; a test pins
the list against the SDK so it doesn’t silently fall behind.
Default: TypeAliasType('KnownOpenAIRealtimeVoiceName', Literal['alloy', 'ash', 'ballad', 'cedar', 'coral', 'echo', 'marin', 'sage', 'shimmer', 'verse'])