Skip to content
Docs Pydantic Docs

pydantic_ai.realtime.elevenlabs

The ElevenLabs realtime provider, for the Eleven Realtime speech-to-speech models. Requires the elevenlabs-realtime optional group (pip install "pydantic-ai-slim[elevenlabs-realtime]"), which includes openai because the model reuses the OpenAI Realtime codec, whose event types come from the OpenAI SDK.

ElevenLabsRealtimeModel is an OpenAIRealtimeModel pointed at ElevenLabs, authenticated by the same ElevenLabsProvider as ElevenLabs Agents. For a hosted ElevenLabs agent, use ElevenLabsAgentsRealtimeModel instead.

ElevenLabs realtime API provider for Eleven Realtime speech-to-speech sessions.

Connects to wss://api.elevenlabs.io/v1/convai/realtime over a WebSocket. Eleven Realtime speaks the OpenAI Realtime protocol, so ElevenLabsRealtimeModel is an OpenAIRealtimeModel pointed at ElevenLabs: it reuses the OpenAI codec (event mapping, session seeding, tool conversion, reconnect-by-replay, and the WebSocket connection itself), authenticated with the key and base URL of the shared ElevenLabsProvider, and diverges only where ElevenLabs does:

  • authentication uses the xi-api-key header, and the model is chosen with session.model in session.update (the server ignores OpenAI’s ?model= query parameter);
  • turn detection is semantic VAD only: the server rejects server_vad, so the cross-provider turn_detection setting maps onto semantic VAD;
  • input transcription is always on, even with input_transcription_model=None, and takes no model, only a language (see elevenlabs_language);
  • voices are ElevenLabs voices, set with elevenlabs_voice;
  • response.done carries no usage, so sessions report no token counts;
  • conversation.item.delete is not implemented, so openai_prune_tool_items is ignored;
  • images are dropped from user messages, and seeded history can only carry text;
  • text output (output_modalities: ['text']) ignores audio input altogether, so output_modality='text' is unsupported;
  • the server ends a session after max_duration_seconds (600 by default; see elevenlabs_max_duration_seconds) or after 60 seconds without a client message. With a reconnect policy, either close is re-dialed and the local message history replayed into a new session.

WebRTC is not supported, as ElevenLabs serves no /realtime/calls endpoint, but create_client_secret mints the ephemeral credential a browser uses to open the WebSocket directly.

For a hosted ElevenLabs agent, use ElevenLabsAgentsRealtimeModel instead.

Requires the websockets package (the realtime optional group) and openai (the openai group, whose SDK supplies the event types the shared OpenAI codec is built on):

pip install “pydantic-ai-slim[elevenlabs-realtime]“

ElevenLabsRealtimeModelSettings

Bases: OpenAIRealtimeModelSettings

Settings specific to Eleven Realtime models.

This inherits every OpenAIRealtimeModelSettings field, since the session is configured with OpenAI’s session.update shape. openai_turn_detection is sent as on OpenAI, except that ElevenLabs only supports semantic VAD: a ServerVAD value raises a UserError.

The remaining inherited fields are silently ignored, because ElevenLabs accepts and then drops them: openai_voice (ElevenLabs voices are set with elevenlabs_voice), openai_input_noise_reduction, openai_output_speed, openai_truncation, and openai_prune_tool_items and openai_keep_tool_items (the server doesn’t implement conversation.item.delete). So are the shared parallel_tool_calls and thinking settings. Input audio is always transcribed: input_transcription_model takes 'auto' (the default) or None, which still leaves transcripts on, and any other value raises a UserError.

Attributes

elevenlabs_voice

Voice used for audio output: an ElevenLabs voice ID (e.g. 'JBFqnCBsd6RMkjVDRZzb') or the name of a default voice. The server’s default is 'brian'.

Type: str

elevenlabs_language

Language code of the conversation (e.g. 'en', 'es', 'pt-br'), used for transcription, speech, and the model’s reply language. The server’s default is English.

Type: str

elevenlabs_strip_audio_tags

Whether to remove expressive audio tags such as [cheerful] from the model’s transcripts. Defaults to True.

Eleven Realtime writes tags into its reply to direct the delivery, and they are part of the transcript, though they aren’t spoken. They are removed from the transcript, streamed and in history alike, and the tags a response’s transcript held are kept, in order and without brackets, in its provider_details['audio_tags']. Only lowercase words in square brackets count as tags, so citations ([1]), Markdown links, and placeholders like [Your Name] are kept. Set this to False to keep the transcript exactly as ElevenLabs sends it, tags included.

Type: bool

elevenlabs_max_duration_seconds

Maximum wall-clock duration of a session, in seconds, from 60 to 7200. The server’s default is 600.

The server closes the WebSocket once it is exceeded. With a reconnect policy, the session carries on over a new connection, with the local message history replayed into it.

Type: int

ElevenLabsRealtimeConnection

Bases: OpenAIRealtimeConnection

A live WebSocket connection to the ElevenLabs realtime API.

Reuses OpenAIRealtimeConnection for the shared wire protocol, including reconnecting by replaying the local message history.

ElevenLabsRealtimeModel

Bases: OpenAIRealtimeModel

ElevenLabs realtime API model, for the Eleven Realtime speech-to-speech models.

Pass provider='elevenlabs' (the default, which reads ELEVENLABS_API_KEY) or an ElevenLabsProvider constructed with api_key= or a data-residency base_url=. The session is configured and driven through the OpenAI Realtime protocol over a WebSocket; WebRTC is not supported.

Constructor Parameters

model : ElevenLabsRealtimeModelName

The model name, eleven_realtime_v1_mini (optimized for price and speed) or eleven_realtime_v1_max (reasons before it responds).

provider : Provider[AsyncHTTPClient] | str Default: 'elevenlabs'

The provider to use for authentication and the base URL. Defaults to 'elevenlabs'.

settings : RealtimeModelSettings | None Default: None

Model settings used as defaults for realtime sessions. With a reconnect policy, a dropped connection is re-dialed and the local message history replayed into it.

profile : RealtimeModelProfileSpec | None Default: None

Optional override for the realtime model profile, merged over the provider’s — a partial dict, or a callable taking the resolved profile and returning the one to use.

LatestElevenLabsRealtimeModelNames

The latest Eleven Realtime model names.

Default: Literal['eleven_realtime_v1_mini', 'eleven_realtime_v1_max']

ElevenLabsRealtimeModelName

Possible Eleven Realtime model names.

eleven_realtime_v1_mini is optimized for price and speed; eleven_realtime_v1_max reasons before it responds, for complex instructions and multi-step tool use, and needs a max_tokens of at least 1024.

Default: str | LatestElevenLabsRealtimeModelNames