ElevenLabs

Pydantic AI connects to ElevenLabs through ElevenLabs Agents.

ElevenLabs Agents

Use an elevenlabs-agents:<agent_id> model string, or ElevenLabsAgentsRealtimeModel for model-level configuration, to connect an agent to the ElevenLabs Agents platform, where the conversation runs on a hosted agent. Start with the realtime quickstart.

ElevenLabs Agents runs a cascaded pipeline (realtime ASR, a configurable LLM, ElevenLabs TTS, and a server-side turn-taking model) behind a pre-provisioned agent, which the conversation reaches over the agent WebSocket. The model wraps that hosted agent, addressed by its agent_id, and Pydantic AI takes ownership of as much of the conversation as ElevenLabs allows per conversation (see Configuration ownership).

Setup

To use ElevenLabs Agents, install pydantic-ai-slim with the elevenlabs-agents-realtime optional group, which bundles the realtime WebSocket transport:

pip

Terminal

pip install "pydantic-ai-slim[elevenlabs-agents-realtime]"

uv

Terminal

uv add "pydantic-ai-slim[elevenlabs-agents-realtime]"

Set ELEVENLABS_API_KEY, or pass an ElevenLabsProvider with api_key=. The key is required even for public agents, because the connect-time preflight authenticates with it. The provider’s base_url doubles as the region switch: pass a data-residency host such as https://api.eu.residency.elevenlabs.io (or the India and Singapore hosts, api.in.residency.elevenlabs.io and api.sg.residency.elevenlabs.io) to keep both the REST calls and the WebSocket inside a region.

Connecting starts with a REST preflight that fetches the agent: a bad key or an unknown agent id raises a ModelHTTPError from it, and an API that can’t be reached raises a RealtimeError. With elevenlabs_agents_tool_sync set to 'error' (the default) or 'off', the key only reads; 'sync' also needs write access to agents and tools.

Before you connect

1. Create the agent in the ElevenLabs dashboard or through the ElevenLabs API, and copy its id.
2. Under Security > Overrides in the agent’s settings, turn on System prompt if the Pydantic AI agent has instructions, plus the toggle of each override setting you use (see Settings).
3. Under Advanced > Client events, add context_usage, which reports context usage and marks where a spoken response ends (see Usage and cost).
4. Pick a tool synchronization mode: with the default, 'error', a difference between the session’s tools and the agent’s client tools fails the connect.

This agent keeps the ElevenLabs agent’s client tools in sync with its own:

elevenlabs_agents_tools.py

from pydantic_ai import Agent
from pydantic_ai.realtime.elevenlabs_agents import ElevenLabsAgentsRealtimeModelSettings

agent = Agent(instructions='You are a friendly concierge. Keep answers short.')


@agent.tool_plain
async def find_restaurant(cuisine: str, party_size: int) -> str:
    """Find a restaurant with a free table.

    Args:
        cuisine: The kind of food the guests want, e.g. 'Italian'.
        party_size: How many guests need a seat.
    """
    return f'Trattoria Roma has a table for {party_size} ({cuisine}).'


realtime_agent = agent.realtime(
    'elevenlabs-agents:agent_0101k2example',
    model_settings=ElevenLabsAgentsRealtimeModelSettings(elevenlabs_agents_tool_sync='sync'),
)

ElevenLabs needs a description for every tool parameter, which the Args: section of the docstring provides.

Model names

Use elevenlabs-agents:<agent_id> as a model string, for example elevenlabs-agents:agent_0101k2..., or pass the agent id, agent_0101k2..., to ElevenLabsAgentsRealtimeModel. The agent’s own configuration decides the ASR setup, audio formats, turn-taking, and which per-conversation overrides are permitted. Every ModelResponse is stamped with the agent id as its model_name; the LLM the agent runs on is part of the agent configuration and is not reported per response.

Configuration ownership

Pydantic AI is the source of truth wherever ElevenLabs allows a per-conversation override, and fails loudly where it does not:

• Instructions: when the Pydantic AI agent defines instructions, the ElevenLabs agent must permit the System prompt override; otherwise the preflight raises a UserError naming the exact toggle. Without local instructions, the ElevenLabs-side prompt is inherited silently.
• Overrides: first message, language, voice, TTS knobs, the pipeline LLM, and text-only mode are sent in the conversation-initiation override payload. Every override is gated by a per-field toggle under Security > Overrides in the agent’s settings (platform_settings.overrides via the API); the preflight checks each override setting against the agent’s allowlist before dialing. The raw elevenlabs_agents_config_override escape hatch is the one exception; see Settings.
• Tools: client tools are workspace entities referenced by the agent, so they cannot be declared inline per conversation. By default the preflight errors when the session’s tools and the agent’s client tools differ in any way; see Tool synchronization.
• Stays with the agent: turn-taking and VAD, ASR configuration, audio formats, and platform settings (auth, privacy, guardrails, knowledge bases, workflows).

Settings

ElevenLabsAgentsRealtimeModelSettings — the realtime counterpart of model run settings — extends the shared settings with the agent’s per-conversation overrides (greeting, language, voice, TTS delivery, the pipeline LLM), dynamic variables, and the tool synchronization mode:

from pydantic_ai.realtime.elevenlabs_agents import (
    ElevenLabsAgentsRealtimeModel,
    ElevenLabsAgentsRealtimeModelSettings,
)

settings = ElevenLabsAgentsRealtimeModelSettings(
    elevenlabs_agents_voice_id='EXAVITQu4vr4xnSDxMaL',
    elevenlabs_agents_language='de',
    elevenlabs_agents_first_message='Hallo! Wie kann ich helfen?',
    elevenlabs_agents_tts={'stability': 0.4, 'speed': 1.05},
)
model = ElevenLabsAgentsRealtimeModel('agent_0101k2example', settings=settings)

Setting	Purpose
elevenlabs_agents_first_message, elevenlabs_agents_language	The greeting the agent opens with and the conversation language
elevenlabs_agents_voice_id, elevenlabs_agents_tts	The provider voice and the TTS delivery knobs (model, stability, speed, similarity boost)
elevenlabs_agents_llm	The LLM the ElevenLabs pipeline runs, by its ElevenLabs model id
elevenlabs_agents_dynamic_variables	Values substituted into {{placeholders}} in the agent’s prompts and messages
elevenlabs_agents_custom_llm_extra_body	Extra request body for an agent backed by a custom LLM; gated by its own security toggle
elevenlabs_agents_user_id	Opaque end-user identifier for ElevenLabs-side analytics
elevenlabs_agents_strip_audio_tags	Whether expressive audio tags such as [laughs] are removed from the agent’s transcripts (True by default); see Provider-specific quirks
elevenlabs_agents_tool_sync	How the session’s tools are reconciled with the agent’s client tools; see Tool synchronization
elevenlabs_agents_config_override	Raw override keys merged last as a forward-compatibility escape hatch

Each elevenlabs_agents_* override setting requires its toggle on the agent, as described above; elevenlabs_agents_dynamic_variables, elevenlabs_agents_user_id, elevenlabs_agents_strip_audio_tags, and elevenlabs_agents_tool_sync need none. elevenlabs_agents_config_override is the one setting deliberately not checked against the permission allowlist — being raw is its point — so a field the agent does not permit there surfaces as the server closing the socket (code 1008, naming the field) after the handshake, not as a local UserError.

Of the shared settings:

• output_modality='text' maps to the text-only override, toggle-gated like the rest.
• tool_choice='none' and ToolOrOutput allow-lists restrict the conversation to those tools through the agent’s Tool IDs override, toggle-gated like the rest: the agent keeps every tool of the run attached, and the conversation leaves the others out. A choice that forces a tool call ('required', or a list of tool names) raises, as on the other providers.
• turn_detection cannot be configured: ElevenLabs’ server-side turn model is always on, so any value other than True raises. There is no push-to-talk.
• input_transcription_model=None raises: ASR drives the agent pipeline and cannot be disabled.
• reconnect raises: a dropped connection ends the session, and a reconnect policy would silently start a conversation that remembers nothing.
• max_tokens, parallel_tool_calls, async_tool_calls, and thinking have no per-conversation surface and are ignored, matching the shared settings contract.

Audio formats

Audio formats are fixed per agent, not per conversation. The model profile defaults to ElevenLabs’ 16 kHz PCM defaults; the connect handshake echoes the agent’s actual formats, and the model raises a RealtimeError when they disagree with the profile. For an agent configured differently (e.g. 24 kHz), pass the actual rates via profile=:

from pydantic_ai.realtime.elevenlabs_agents import ElevenLabsAgentsRealtimeModel

model = ElevenLabsAgentsRealtimeModel(
    'agent_0101k2example',
    profile={'audio_input_sample_rate': 24000, 'audio_output_sample_rate': 24000},
)

Telephony (ulaw_8000) agents are not supported.

Tool synchronization

elevenlabs_agents_tool_sync controls how the session’s ToolDefinitions are reconciled with the agent’s client tools at connect time:

• 'error' (default): raise a UserError describing every difference (missing, extra, or differing tools), so the two can never silently disagree.
• 'sync': make the workspace match Pydantic AI over REST before dialing: create missing client tools, update differing ones, and re-point the agent’s tool_ids. A differing tool that another agent also uses isn’t changed under it: this agent gets a copy of its own. This mutates workspace state shared by every conversation with the agent, which is why it is opt-in. Treat it as a deploy-time step — a warm-up connect after a tool change — rather than something a customer’s first call should pay for: the writes run one after another, and two processes syncing the same agent concurrently race on the final tool_ids re-point, last writer wins. Synced tools get ElevenLabs’ default 20-second response timeout (response_timeout_secs); a tool that already has a timeout of its own keeps it when it is updated.
• 'off': trust the agent’s own tool definitions, e.g. for client tools set up on the ElevenLabs side with settings Pydantic AI doesn’t sync. A call to a tool the session does not define reaches the tool manager like any other call and is answered with the standard unknown-tool error, which lands in the session’s history.

Server-side webhook, MCP, and system tools on the agent are ElevenLabs-owned and never touched, and ElevenLabs runs them itself. Its calls to webhook and system tools (such as end_call) aren’t recorded in the session’s history; its calls to MCP tools are recorded as a NativeToolCallPart and NativeToolReturnPart. MCP tools that require approval on the ElevenLabs side aren’t supported: the session denies such a call (with a warning) rather than leave the conversation waiting. Give the agent those tools through a Pydantic AI MCP toolset instead, which is the recommended way, or turn approval off for the tool in the agent’s configuration.

ElevenLabs stores tool parameters in its own restricted schema dialect rather than JSON Schema. type, description, enum, items, properties, and required are supported; other keywords (such as additionalProperties or numeric bounds) are not; and every parameter needs a description. Nested models ($defs/$ref) are inlined, and optional (nullable) parameters are collapsed onto their non-null type before conversion, with optionality carried by required alone. The dialect has no way to say null — a required nullable parameter is advertised as its non-null type, and the agent’s LLM, which only ever sees the advertised schema, always supplies a value for it. 'sync' therefore fails loudly for a tool whose parameters lack descriptions (add them in the function docstring, and set require_parameter_descriptions=True to catch a missing one when the tool is defined) or use a shape the dialect cannot express (a union of types, or a recursive reference). The 'error' comparison checks what the dialect can express, comparing a parameter’s description only when the local schema declares one.

Feature support and limitations

Feature	Support	Notes
Audio format	Full feature support	Mono PCM16; rates are fixed per agent (16 kHz default) and validated at connect, see Audio formats
Text output	Limited parameter support	output_modality='text' maps to the text-only override, which needs its toggle on the agent
Image input	Unsupported	send() rejects images with a UserError
Manual turns	Unsupported	ElevenLabs’ server-side turn model owns turn-taking; turn_detection must stay True
Explicit interruption/truncation	Unsupported	ElevenLabs interrupts server-side and emits RealtimeResponseInterruptedEvent; interrupt() raises
Session seeding	Unsupported	The conversation WebSocket has no history-seeding channel
Input transcription	Full feature support	Always on: per-utterance final transcripts from the agent’s ASR; input_transcription_model=None raises
Native tools	Unsupported	None can be configured from the session; configure local fallbacks for web capabilities. The agent’s own MCP tool calls are recorded as NativeToolCallParts named mcp_server:<server id>
Async tool calls	Unsupported	The agent waits for each tool result (see concurrent tool execution)
Usage	Limited parameter support	Context tokens only, with context_usage in the agent’s client_events; no cost (see Usage and cost)
Context window	Limited parameter support	Known only once the agent reports context_usage: session.context_window_used then gives the share of the LLM’s context in use (see Usage and cost)
Session duration	Limited parameter support	Capped by the agent’s maximum conversation duration (600 s by default, up to 7,200 s, set on the agent, not per conversation); at the cap the session gets a non-recoverable RealtimeSessionErrorEvent and raises a RealtimeError
State-restoring reconnect	Unsupported	A dropped connection ends the session, and reconnect raises, see Provider session limits

See Audio, images, and transcripts, Turns and interruptions, Tools, and Connection lifecycle for the provider-agnostic workflows.

Gateway

ElevenLabs is not currently available through the Pydantic AI Gateway. Connect with an elevenlabs-agents:<agent_id> model string, or with provider='elevenlabs' or an ElevenLabsProvider.

Usage and cost

ElevenLabs reports the LLM’s context consumption only, as input_tokens, and only when context_usage is in the agent’s conversation.client_events list (it is not by default, and adding it is recommended: it also tells the session where a spoken response ends, see Provider-specific quirks; a session on an agent without it warns once). A report arrives with the end of each response’s audio and counts toward the session’s usage without belonging to any one response. When ElevenLabs answers a turn with a fallback LLM, the report names that model and its context window. The context window it names is kept on ElevenLabsAgentsRealtimeConnection.context_limit_tokens, and context_window_used is the ratio of the two.

No price is known for the platform, so usage.cost is always None and a cost_limit can’t be enforced. Look the cost up afterwards through the ElevenLabs conversations API: every ModelResponse the conversation ends, cut off and lost ones included, carries the conversation’s id in provider_details['conversation_id'] (one still under way when the session itself closes has none), and the live connection has it as ElevenLabsAgentsRealtimeConnection.conversation_id.

Provider-specific quirks

• In audio mode, a response’s transcript arrives whole once the agent’s LLM has finished, which can be before or during its audio; only text-only mode streams it. Synthesis outruns playback, so a spoken response is usually delivered in full while the user is still hearing its first words.
• With a TTS model that voices expressive audio tags, such as eleven_v3_conversational or eleven_v4_turbo, the agent’s LLM writes tags like [laughs] into its reply to direct the delivery, and ElevenLabs sends them as part of the response’s transcript. Pydantic AI removes them, from the streamed transcript and the one in history alike, and keeps them, in order and without brackets, in the response’s provider_details['audio_tags'] (['laughs']). Only lowercase words in square brackets count as tags, so citations ([1]), Markdown links, and placeholders like [Your Name] stay in the text. Set elevenlabs_agents_strip_audio_tags=False to keep the transcript as ElevenLabs sends it ([laughs] Hi there.).
• A barge-in on a response still being generated ends it with the transcript cut down to what the user heard (or, when ElevenLabs sends a correction instead, with the cut-down text in provider_details['corrected_agent_response']). One on a response already delivered keeps its full text in history; ElevenLabs’ correction of it is dropped.
• send(text, respond=False) is a contextual_update: the agent reads it from its next turn on, without stopping or replying.
• A response’s parallel tool calls each run as soon as they arrive.
• With elevenlabs_agents_tool_sync='off', client tools registered on the ElevenLabs side with expects_response: false are fire-and-forget: the session still runs them, but their results are never sent back to the agent. (The other modes require every client tool to expect a response.)
• A conversation waiting in the workspace’s concurrency queue hears ElevenLabs’ hold audio, which isn’t part of the conversation; one that times out there ends with a RealtimeError saying so.