Connecting a frontend

Keep provider keys, tools, and business logic on the server; connect user devices to your backend, not straight to the provider. How audio travels between the device and the model depends on the client and the provider:

• Browser WebRTC + server sideband — the recommended browser path on OpenAI and Azure OpenAI: the browser exchanges media directly with the provider (lowest latency) while your backend runs the agent over a control-plane sideband.
• Browser → backend WebSocket relay — works with every provider, including Gemini Live and xAI: the browser streams audio to your backend, which owns the media bridge.
• SIP / telephony bridge — for phone calls, via a telephony provider.

In every shape the session — with its tools, history, and usage limits — runs on your backend. Wiring a browser straight to the provider with the provider’s own SDK instead moves the agent loop into the client and gives up all of that; prefer the shapes above.

Browser WebRTC + server sideband

For browser voice agents on OpenAI and Azure OpenAI, the browser carries microphone and speaker audio directly over WebRTC while the backend attaches a control-plane sideband to the same call. Media never touches your backend, so latency stays low, while tools, history, dependencies, and provider credentials stay on the server. Gemini Live and xAI do not offer this sideband transport — use the WebSocket relay there.

browser ── WebRTC media ── provider
   │                         ▲
   └─ SDP offer → backend ───┘ sideband identified by call_id

Relay the browser’s offer with AgentRealtime.answer_webrtc_offer, return the SDP answer to the browser, then attach the returned call handle:

import asyncio

from pydantic_ai import Agent

agent = Agent(instructions='You are a concise voice assistant.')
realtime = agent.realtime('openai:gpt-realtime')


async def handle_offer(sdp_offer: str) -> str:
    answer = await realtime.answer_webrtc_offer(sdp_offer)

    async def run_sideband() -> None:
        async with realtime.session(provider_session=answer.session) as session:
            async for event in session:
                print(event)

    asyncio.create_task(run_sideband())
    return answer.sdp

The secure offer-relay flow never gives the browser a token. As an alternative, AgentRealtime.create_client_secret mints a short-lived credential for client-led negotiation. Either way the browser is a peer on the provider session and can send provider-native control events, so authorize every server-side tool against trusted deps, not session instructions supplied to the model.

The browser can read seeded history

Seeding a sideband session with message_history sends those prior turns into the shared provider conversation that the browser is a peer on, so a call participant can read them — including confidential tool results — over the data channel (Azure’s webrtcfilter=on still forwards conversation-item events). Only seed a sideband with history that is safe for the browser to see; keep confidential context in deps and tool logic instead.

A sideband does not own the audio transport: its send_audio(), commit_audio(), clear_audio(), and stream_audio() methods raise, and audio_retention must remain 'transcript_only'. Enable input transcription when user speech must appear in history, and use RealtimeOutputSpeechStartEvent / RealtimeOutputSpeechEndEvent for speaking indicators — interrupt() clears the provider’s outbound WebRTC audio buffer so barge-in stops playback. A dropped sideband follows the same reconnect rules, with one wrinkle covered there: a clean close is treated as the browser hanging up.

The realtime WebRTC example demonstrates the full FastAPI and browser flow. Provider-specific setup (Azure’s Microsoft Entra ID and webrtcfilter) lives on the Azure page.

Browser → backend WebSocket relay

When the browser can’t use WebRTC — or the provider is Gemini Live or xAI — build a WebSocket endpoint on your backend that accepts the browser’s microphone audio and pumps it into send_audio(), while relaying stream_audio() output back for playback. Here the backend owns the media bridge. The realtime camera example demonstrates this shape end to end; a minimal FastAPI relay — the browser sends raw PCM16 binary frames and plays the frames it receives — is:

import asyncio

from fastapi import FastAPI, WebSocket

from pydantic_ai import Agent

agent = Agent(instructions='You are a helpful voice assistant.')
app = FastAPI()


@app.websocket('/voice')
async def voice_socket(websocket: WebSocket):
    await websocket.accept()
    async with agent.realtime('openai:gpt-realtime').session() as session:

        async def pump_input():
            while True:
                await session.send_audio(await websocket.receive_bytes())

        input_task = asyncio.create_task(pump_input())
        try:
            async for chunk in session.stream_audio():
                await websocket.send_bytes(chunk)
        finally:
            input_task.cancel()

SIP/telephony bridge

Terminate the phone call with a telephony provider such as Twilio, then build the service that connects its media stream (e.g. Twilio Media Streams over WebSocket) to the backend session, transcoding between the line’s codec and PCM16.