pydantic_ai.realtime.elevenlabs_agents
The ElevenLabs Agents realtime provider. Requires the elevenlabs-agents-realtime optional group
(pip install "pydantic-ai-slim[elevenlabs-agents-realtime]").
ElevenLabs Agents provider for realtime voice sessions.
ElevenLabs Agents runs a cascaded pipeline (realtime ASR + a configurable LLM + ElevenLabs TTS + a
server-side turn-taking model) behind a pre-provisioned agent, reached over the agent WebSocket
(wss://api.elevenlabs.io/v1/convai/conversation?agent_id=...). So unlike the other realtime
providers, the model here wraps a hosted agent addressed by its agent_id, and Pydantic AI takes
ownership of as much of the conversation as ElevenLabs allows per-conversation:
- Instructions, LLM choice, first message, language, voice, TTS knobs, and text-only mode are pushed
through
conversation_initiation_client_data.conversation_config_override. Every overridable field is gated by a per-field toggle under Security > Overrides in the agent’s settings (platform_settings.overrides); the connect-time preflight fetches the agent and raises aUserErrornaming the exact toggle when an override the session needs is not permitted. (The rawelevenlabs_agents_config_overrideescape hatch skips this check by design and is judged by the server after the handshake instead.) When the Pydantic AI agent defines no instructions, the ElevenLabs-side prompt is inherited silently. - Tool declarations cannot be sent inline per-conversation: client tools are workspace entities
referenced from the agent. By default the preflight errors on mismatch between the session’s
ToolDefinitions and the agent’s client tools; the opt-inelevenlabs_agents_tool_sync='sync'setting instead creates/updates the workspace tools and re-points the agent’stool_idsover REST before dialing. - Turn-taking, VAD, ASR configuration, audio formats, and platform settings (auth, privacy, guardrails, knowledge bases) stay owned by the ElevenLabs agent configuration.
Requires the websockets package, available via the elevenlabs-agents-realtime optional group (REST
preflight uses Pydantic AI’s own HTTP client, so no ElevenLabs SDK is needed):
pip install “pydantic-ai-slim[elevenlabs-agents-realtime]“
Bases: TypedDict
Per-conversation TTS overrides, each gated by its own override toggle on the agent.
The ElevenLabs TTS model to synthesize with, e.g. eleven_v3_conversational.
Type: str
Voice stability (0-1); lower is more expressive, higher more consistent.
Type: float
Playback speed multiplier.
Type: float
Voice similarity boost (0-1).
Type: float
Bases: RealtimeModelSettings
Settings specific to ElevenLabs Agents realtime sessions.
Every elevenlabs_agents_* field below that maps to a conversation_config_override entry requires the
matching override toggle to be enabled under Security > Overrides in the agent’s settings; the connect-time
preflight raises UserError naming the toggle otherwise.
Of the shared settings, output_modality='text' maps to the conversation.text_only override, and
a tool_choice of 'none' or an allow-list to the agent.prompt.tool_ids override, which narrows
the agent’s attached tools for the conversation (a choice that forces a tool call raises). max_tokens,
parallel_tool_calls, async_tool_calls, and thinking have no per-conversation surface on the
agent WebSocket and are silently ignored, per the shared settings contract. turn_detection cannot be configured
(ElevenLabs’ server-side turn model is always on), input_transcription_model=None cannot be
honored (ASR cannot be disabled per-conversation), and reconnect is unsupported (ElevenLabs’
experimental, text-only resume isn’t used), so each of those raises rather than silently under-delivering.
Voice used for audio output (the tts.voice_id override).
Type: str
Conversation language code, e.g. de (the agent.language override).
Type: str
The greeting the agent opens with (the agent.first_message override).
Type: str
The LLM the ElevenLabs agent pipeline runs, e.g. gpt-5.2 (the agent.prompt.llm override).
Type: str
TTS model and voice-delivery knobs (the tts.* overrides).
Type: ElevenLabsAgentsTTSSettings
Values substituted into {{placeholders}} in the agent’s prompts and messages.
Type: dict[str, str | int | float | bool]
Extra body merged into the agent’s custom-LLM request. Gated by the agent’s
custom_llm_extra_body security toggle (a sibling of the override toggles).
Opaque end-user identifier attached to the conversation for ElevenLabs-side analytics.
Type: str
How to reconcile the session’s tools with the agent’s client tools at connect time.
'error'(default): raiseUserErrordescribing every difference between the session’sToolDefinitions and the agent’s client tools, so the two can never silently disagree.'sync': make the workspace match Pydantic AI over REST before dialing: create missing client tools, update differing ones, and re-point the agent’stool_ids(dropping client tools the session doesn’t define; ElevenLabs-side webhook/MCP/system tools are left untouched). This mutates workspace state shared by every conversation with the agent, which is why it is opt-in.'off': skip the check and trust the agent’s own tool definitions, e.g. for client tools configured on the ElevenLabs side with settings Pydantic AI doesn’t sync. A call to a tool the session does not define reaches the tool manager like any other call and is answered with the standard unknown-tool error, which lands in the session’s history.
Type: Literal[‘error’, ‘sync’, ‘off’]
Whether to remove expressive audio tags such as [laughs] from the agent’s transcripts. Defaults to True.
With an expressive TTS model such as eleven_v3_conversational, the agent’s LLM writes tags into its reply to
direct the delivery, and they are part of the transcript ElevenLabs sends, though they aren’t spoken. They are
removed from the transcript, streamed and in history alike, and the tags a response’s transcript held are kept,
in order and without brackets, in its provider_details['audio_tags']. Only lowercase words in square brackets
count as tags, so citations ([1]), Markdown links, and placeholders like [Your Name] are kept. Set this to
False to keep the transcript exactly as ElevenLabs sends it, tags included.
Type: bool
Raw values deep-merged last into conversation_config_override, the escape hatch mirroring
google_config_overrides. Not preflight-checked against the override toggles; the server still
rejects non-permitted fields.
Bases: RealtimeModel
ElevenLabs Agents realtime model, wrapping a hosted agent addressed by its agent_id.
agent_id : str
The ElevenLabs agent to converse with, e.g. agent_0101k2.... The agent must exist;
Pydantic AI overrides its per-conversation configuration where the agent’s override
toggles allow, and fails loudly where they don’t.
provider : Provider[AsyncHTTPClient] | str Default: 'elevenlabs'
The provider to use for authentication and API access. Defaults to 'elevenlabs'
(reads ELEVENLABS_API_KEY); pass an
ElevenLabsProvider for a custom
key, HTTP client, or regional (data-residency) base URL.
settings : RealtimeModelSettings | None Default: None
Model settings used as defaults for realtime sessions.
profile : RealtimeModelProfileSpec | None Default: None
Optional override for the realtime model profile, merged over the provider’s. For ElevenLabs this is how per-agent audio formats are declared when the agent isn’t on the default 16 kHz PCM formats.
The agent id; ElevenLabs has no realtime model id apart from the agent.
Type: str
Bases: RealtimeConnection
A live WebSocket connection to an ElevenLabs Agents conversation.
Always None: ElevenLabs reports no realtime model id, so responses carry the agent id.
The LLM behind the agent only surfaces in the context_usage report, which an agent sends only with it in
its client_events, so it would name the model of some responses and not others, and the session
falls back to ElevenLabsAgentsRealtimeModel.model_name.
The server-assigned conversation id, e.g. for post-hoc cost lookup via the conversations API.
The context window of the LLM behind the agent, as last reported by a context_usage event.
None until the agent reports one, which requires context_usage in its
conversation.client_events list. It is a property of the conversation, not of a response,
so it is kept here rather than summed into the session’s usage, and the session’s
context_window_used is only known once such a
report arrives.
@async
def send(content: RealtimeInput) -> Sequence[LifecycleEvent]
Send content to the ElevenLabs Agents conversation.
Accepts BinaryAudio (raw PCM16 mono at the agent’s input rate), a str text turn,
TextContext (a contextual_update: context the agent reads from its next turn on, without
interrupting or answering), and ToolResult. Manual turn-taking verbs, response
cancellation, truncation, and images are not supported by the conversation WebSocket.
Sequence[LifecycleEvent]