Multimodal Input

Alongside text, agents can accept image, audio, video, and document input, as long as the model supports it.

Image Input

Note

Some models do not support image input. Please check the model’s documentation to confirm whether it supports image input.

If you have a direct URL for the image, you can use ImageUrl:

image_input.py

from pydantic_ai import Agent, ImageUrl

agent = Agent(model='openai:gpt-6.1-sol')
result = agent.run_sync(
    [
        'What company is this logo from?',
        ImageUrl(url='https://iili.io/3Hs4FMg.png'),
    ]
)
print(result.output)
#> This is the logo for Pydantic, a data validation and settings management library in Python.

If you have the image locally, you can also use BinaryContent:

local_image_input.py

import httpx

from pydantic_ai import Agent, BinaryContent

image_response = httpx.get('https://iili.io/3Hs4FMg.png')  # Pydantic logo

agent = Agent(model='openai:gpt-6.1-sol')
result = agent.run_sync(
    [
        'What company is this logo from?',
        BinaryContent(data=image_response.content, media_type='image/png'),  # (1)
    ]
)
print(result.output)
#> This is the logo for Pydantic, a data validation and settings management library in Python.

To ensure the example is runnable we download this image from the web, but you can also use Path().read_bytes() to read a local file's contents.

Audio Input

Note

Some models do not support audio input. Please check the model’s documentation to confirm whether it supports audio input.

For audio, use AudioUrl or BinaryContent.

Video Input

Note

Some models do not support video input. Please check the model’s documentation to confirm whether it supports video input.

For video, use VideoUrl or BinaryContent.

Document Input

Note

Some models do not support document input. Please check the model’s documentation to confirm whether it supports document input.

For documents, use DocumentUrl or BinaryContent.

If you have a direct URL for the document, you can use DocumentUrl:

document_input.py

from pydantic_ai import Agent, DocumentUrl

agent = Agent(model='anthropic:claude-sonnet-5-5')
result = agent.run_sync(
    [
        'What is the main content of this document?',
        DocumentUrl(url='https://storage.googleapis.com/cloud-samples-data/generative-ai/pdf/2403.05530.pdf'),
    ]
)
print(result.output)
#> This document is the technical report introducing Gemini 1.5, Google's latest large language model...

The supported document formats vary by model.

You can also use BinaryContent to pass document data directly:

binary_content_input.py

from pathlib import Path
from pydantic_ai import Agent, BinaryContent

pdf_path = Path('document.pdf')
agent = Agent(model='anthropic:claude-sonnet-5-5')
result = agent.run_sync(
    [
        'What is the main content of this document?',
        BinaryContent(data=pdf_path.read_bytes(), media_type='application/pdf'),
    ]
)
print(result.output)
#> The document discusses...

Tip

If neither DocumentUrl nor BinaryContent is suitable for your use case (e.g., the model doesn’t support DocumentUrl, or you want to provide a document in a non-binary format), you can still provide document content as text input by extracting the text yourself and passing it as a string or TextContent.

Text Input

You can use TextContent to provide text input with additional metadata:

text_content_input.py

from pydantic_ai import Agent, TextContent

agent = Agent(model='openai:gpt-6.1-sol')
result = agent.run_sync([
    'Summarize the key points from this text.',
    TextContent(
        content=(
            'Pydantic AI is a Python agent framework. '
            'It supports text, image, audio, video, and document input.'
        ),
        metadata={'source': 'pydantic_ai_inputs.txt'},
    ),
])

This is equivalent to passing the text as a str, but allows you to include additional metadata that can be accessed programmatically in your agent logic.

Note

The content field is treated as input to the model, but the metadata is not sent to the model. It is preserved in messages for programmatic access.

Structured Input

Use StructuredContent to pass a structured value as part of the user prompt, such as the state of the app the user is looking at, or the data a decision model should judge. The value stays structured in the message history, so a history processor or an evaluator can read it back, while the model is sent it as text:

structured_content_input.py

from pydantic import BaseModel

from pydantic_ai import Agent, StructuredContent


class UIState(BaseModel):
    url: str
    cart_total: float


agent = Agent('openai:gpt-6.1-sol')
ui_state = UIState(url='/products/42', cart_total=949.0)
result = agent.run_sync([StructuredContent(ui_state, tag='ui_state'), 'Why did the total jump?'])

The model is sent <ui_state>{"url":"/products/42","cart_total":949.0}</ui_state>, followed by the question. By default the value is rendered as compact JSON, wrapped in tag when one is given, with <, > and & escaped so that a value from an untrusted client, rendered on your server, can’t close the tag. Pass format='xml' to render it with format_as_xml instead, or a function that takes the value and returns the text. A description is written on the line before the value. A tag is lowercase with _ between words, and can’t be one of the tags the framework frames its own text in, such as system or tool_result, so the value can’t read as the framework speaking.

The text is rendered once, when the item is created, and stored in content beside the value. A message history that is stored and loaded again is therefore sent exactly as before, which keeps the prompt cache prefix intact, even when the text came from a function that isn’t stored. The text isn’t rendered again when the value changes, so to change the value, create a new item, or use dataclasses.replace(item, value=..., content=''), passing format again if the item was created with one. After loading, value is the JSON-compatible form of the value, such as a dict in place of a Pydantic model.

To render a kind of value the same way wherever it is used, subclass StructuredContent and override render:

structured_content_subclass.py

from pydantic import BaseModel

from pydantic_ai import StructuredContent, format_as_xml


class UIState(BaseModel):
    url: str
    cart_total: float


class UIStateContent(StructuredContent):
    value: UIState

    def render(self) -> str:
        return format_as_xml(self.value, root_tag='ui_state')


print(UIStateContent(UIState(url='/products/42', cart_total=949.0)).content)
"""
<ui_state>
  <url>/products/42</url>
  <cart_total>949.0</cart_total>
</ui_state>
"""

StructuredContent is a TextContent, so code that reads the text of a prompt handles it like any other text. A decision model is the exception: it is sent the value itself when the prompt is a single StructuredContent. Code that rewrites the text of a prompt, such as a redaction step, should therefore replace the item with a plain TextContent holding the new text, as the harness’s guardrails and compaction do.

The Vercel AI and AG-UI adapters dump the item as ordinary text in the user message, which a frontend shows like any other user text, and carry its value in the metadata of that text part (Vercel AI) or message (AG-UI, from ag-ui-protocol 0.1.11), so it loads back whole. The same goes for an item enqueued mid-run with ctx.enqueue, which accepts it like any other user content. Like TextContent.metadata, its metadata doesn’t make the round-trip.

User-side download vs. direct file URL

When using one of ImageUrl, AudioUrl, VideoUrl or DocumentUrl, Pydantic AI will default to sending the URL to the model provider, so the file is downloaded on their side.

Support for file URLs varies depending on type and provider:

Model	Send URL directly	Download and send bytes	Unsupported
OpenAIChatModel	ImageUrl	AudioUrl, DocumentUrl	VideoUrl. DocumentUrl not supported with AzureProvider or AlibabaProvider
OpenAIResponsesModel	ImageUrl, AudioUrl, DocumentUrl	—	VideoUrl
AnthropicModel	ImageUrl, DocumentUrl (PDF)	DocumentUrl (text/plain)	AudioUrl, VideoUrl
GoogleModel (Google Cloud)	All URL types	—	—
GoogleModel (Gemini API)	YouTube, Files API	All other URLs	—
XaiModel	ImageUrl	DocumentUrl	AudioUrl, VideoUrl
MistralModel	ImageUrl, DocumentUrl (PDF)	DocumentUrl (text/plain)	AudioUrl, VideoUrl, DocumentUrl (non-PDF, non-text)
BedrockConverseModel	S3 URLs (s3://)	ImageUrl, DocumentUrl, VideoUrl	AudioUrl
OpenRouterModel	ImageUrl, DocumentUrl, VideoUrl	AudioUrl	—

A model that does not accept a kind of file at all can still be given one through the File Understanding capability, which has another model describe it in text first. Without it, a file in a user prompt that the model can’t take fails the request, while one a tool returned is replaced by a note naming it, so the run goes on.

A model API may be unable to download a file (e.g., because of crawling or access restrictions) even if it supports file URLs. For example, GoogleModel on Google Cloud limits YouTube video URLs to one URL per request.

In such cases, you can instruct Pydantic AI to download the file content locally and send that instead of the URL by setting force_download on the URL object:

force_download.py

from pydantic_ai import ImageUrl, AudioUrl, VideoUrl, DocumentUrl

ImageUrl(url='https://example.com/image.png', force_download=True)
AudioUrl(url='https://example.com/audio.mp3', force_download=True)
VideoUrl(url='https://example.com/video.mp4', force_download=True)
DocumentUrl(url='https://example.com/doc.pdf', force_download=True)

When Pydantic AI downloads a file, it sends the media_type you passed if it differs from the one the URL’s extension implies. Otherwise it uses the server’s Content-Type, falling back to the URL’s extension when the header is missing or generic (application/octet-stream or binary/octet-stream). Pass media_type explicitly for URLs without an extension, or for servers that report the wrong type:

explicit_media_type.py

from pydantic_ai import ImageUrl

ImageUrl(url='https://example.com/receipt-123', media_type='image/png', force_download=True)

Trust model for file URLs

When URLs are forwarded to the provider, the provider fetches them under its own credentials. For cloud-storage schemes like s3:// (Bedrock) and gs:// (Google Cloud), those credentials are your server’s IAM role or service account, so whoever controls the URL effectively controls what the provider can read on your behalf.

Don’t construct ImageUrl, AudioUrl, VideoUrl, or DocumentUrl from untrusted user input without validating the scheme and scope. For frontend-initiated uploads to cloud storage, convert references like s3://bucket/key into pre-signed https:// URLs server-side before constructing the file URL part. force_download=True only works for http(s):// URLs (it routes through the library’s HTTP client and applies SSRF protection); cloud-storage schemes like s3:// and gs:// aren’t supported by the local download path and are forwarded to the provider as-is. Only use force_download='allow-local' for server-authored URLs, since it allows local network access.

The UI adapters apply this sanitization automatically to client-submitted messages via UIAdapter.allowed_file_url_schemes and UIAdapter.allowed_file_url_force_download. If you accept serialized message_history through a custom client API, use sanitize_messages before passing that history to the agent.

Uploaded Files

Some model providers have their own file storage APIs where you can upload files and reference them by ID or URL.

Use UploadedFile to reference files that have been uploaded to a provider’s file storage API.

Tip

For providers that return a file URL (like Google Files API or S3 URLs for Bedrock), you can also use DocumentUrl, ImageUrl, or VideoUrl directly. However, we recommend using UploadedFile for a unified API across providers and consistent provider name validation.

Supported Models

Model	Support
AnthropicModel	✅ via Anthropic Files API
OpenAIChatModel	✅ via OpenAI Files API
OpenAIResponsesModel	✅ via OpenAI Files API
GoogleModel	✅ via Google Files API
BedrockConverseModel	✅ via S3 URLs (s3://bucket/key)
XaiModel	✅ via xAI Files API
Other models	❌ Not supported

Provider Name Requirement

When using UploadedFile you must set the provider_name. Uploaded files are specific to the system they are uploaded to and are not transferable across providers. Trying to use a message that contains an UploadedFile with a different provider will result in an error.

Getting the provider name

Use model.system to get the correct provider name dynamically. This ensures your code works correctly even if the provider name changes. All examples below demonstrate this pattern.

If you want to introduce portability into your agent logic to allow the same prompt history to work with different provider backends, you can use a history processor to remove or rewrite UploadedFile parts from messages before sending them to a provider that does not support them. Be aware that stripping out UploadedFile instances might confuse the model, especially if references to those files remain in the text.

Media Type Inference

The media_type parameter is optional for UploadedFile. If not specified, Pydantic AI will attempt to infer it from the file_id:

1. If file_id is a URL or path with a recognizable file extension (e.g., .pdf, .png), the media type is inferred automatically
2. For opaque file IDs (e.g., 'file-abc123'), the media type defaults to 'application/octet-stream'

Tip

While media_type is optional, we recommend explicitly setting it when known to ensure correct handling by the model provider.

Anthropic

Follow the Anthropic Files API docs to upload files. You can access the underlying Anthropic client via provider.client.

Beta Feature

The Anthropic Files API is currently in beta. AnthropicModel automatically adds the required anthropic-beta: files-api-2025-04-14 header when a request contains an Anthropic UploadedFile, so you don’t need to set it yourself.

uploaded_file_anthropic.py

import asyncio

from pydantic_ai import Agent, UploadedFile
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.providers.anthropic import AnthropicProvider


async def main():
    provider = AnthropicProvider()
    model = AnthropicModel('claude-sonnet-5-5', provider=provider)

    # Upload a file using the provider's client (Anthropic client)
    with open('document.pdf', 'rb') as f:
        uploaded_file = await provider.client.beta.files.upload(file=f)

    # Reference the uploaded file; the beta header is added automatically
    agent = Agent(model)
    result = await agent.run(
        [
            'Summarize this document',
            UploadedFile(file_id=uploaded_file.id, provider_name=model.system),
        ]
    )
    print(result.output)
    #> The document discusses the main topics and key findings...


asyncio.run(main())

OpenAI

Follow the OpenAI Files API docs to upload files. You can access the underlying OpenAI client via provider.client.

uploaded_file_openai.py

import asyncio

from pydantic_ai import Agent, UploadedFile
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider


async def main():
    provider = OpenAIProvider()
    model = OpenAIChatModel('gpt-6.1-sol', provider=provider)

    # Upload a file using the provider's client (OpenAI client)
    with open('document.pdf', 'rb') as f:
        uploaded_file = await provider.client.files.create(file=f, purpose='user_data')

    # Reference the uploaded file
    agent = Agent(model)
    result = await agent.run(
        [
            'Summarize this document',
            UploadedFile(file_id=uploaded_file.id, provider_name=model.system),
        ]
    )
    print(result.output)
    #> The document discusses the main topics and key findings...


asyncio.run(main())

Referencing uploaded images

OpenAIChatModel can only reference uploaded documents by file_id. Referencing an uploaded image (an image/* media type) raises a UserError, because the Chat Completions API doesn’t accept a file_id for image parts. Use ImageUrl or BinaryContent for images, or use OpenAIResponsesModel, which does support uploaded images.

With OpenAIResponsesModel, control the image detail level by passing vendor_metadata={'detail': 'high'} (or 'low') to the UploadedFile; it defaults to 'auto'. With OpenAIChatModel, GroqModel, MistralModel, and XaiModel, control the image detail level by passing vendor_metadata={'detail': 'high'} (or 'low') to the ImageUrl or BinaryContent; it defaults to 'auto'.

Google

Follow the Google Files API docs to upload files. You can access the underlying Google GenAI client via provider.client.

uploaded_file_google.py

import asyncio

from pydantic_ai import Agent, UploadedFile
from pydantic_ai.models.google import GoogleModel
from pydantic_ai.providers.google import GoogleProvider


async def main():
    provider = GoogleProvider()
    model = GoogleModel('gemini-3.8-flash', provider=provider)

    # Upload a file using the provider's client (Google GenAI client)
    with open('document.pdf', 'rb') as f:
        file = await provider.client.aio.files.upload(file=f)
        assert file.uri is not None

    # Reference the uploaded file by URI (media_type is optional for Google)
    agent = Agent(model)
    result = await agent.run(
        [
            'Summarize this document',
            UploadedFile(file_id=file.uri, media_type=file.mime_type, provider_name=model.system),
        ]
    )
    print(result.output)
    #> The document discusses the main topics and key findings...


asyncio.run(main())

Bedrock (S3)

For Bedrock, files must be uploaded to S3 separately (e.g., using boto3). The assumed role must have s3:GetObject permission on the bucket.

media_type may be required

Bedrock requires media_type when the file extension is ambiguous or missing. For S3 URLs with clear extensions like .pdf, .png, etc., it can be inferred automatically.

uploaded_file_bedrock.py

import asyncio

from pydantic_ai import Agent, UploadedFile
from pydantic_ai.models.bedrock import BedrockConverseModel


async def main():
    model = BedrockConverseModel('global.anthropic.claude-sonnet-5-5')

    agent = Agent(model)
    result = await agent.run([
        'Summarize this document',
        UploadedFile(
            file_id='s3://my-bucket/document.pdf',
            provider_name=model.system,  # 'bedrock'
            media_type='application/pdf',  # Optional for .pdf, but recommended
        ),
    ])
    print(result.output)
    #> The document discusses the main topics and key findings...


asyncio.run(main())

Note

You can optionally specify a bucketOwner query parameter if the bucket is not owned by the account making the request: s3://my-bucket/document.pdf?bucketOwner=123456789012

xAI

Follow the xAI Files API docs to upload files. You can access the underlying xAI client via provider.client.

uploaded_file_xai.py

import asyncio

from pydantic_ai import Agent, UploadedFile
from pydantic_ai.models.xai import XaiModel
from pydantic_ai.providers.xai import XaiProvider


async def main():
    provider = XaiProvider()
    model = XaiModel('grok-4.7', provider=provider)

    # Upload a file using the provider's client (xAI client)
    with open('document.pdf', 'rb') as f:
        uploaded_file = await provider.client.files.upload(f, filename='document.pdf')

    # Reference the uploaded file
    agent = Agent(model)
    result = await agent.run(
        [
            'Summarize this document',
            UploadedFile(file_id=uploaded_file.id, provider_name=model.system),
        ]
    )
    print(result.output)
    #> The document discusses the main topics and key findings...


asyncio.run(main())