Batch Models
The data types of a provider’s batch API, for batch pricing in exchange for latency.
The BatchModelRequests capability lets an agent use a batch API
without changing how it runs, and model_request_batch sends a list of
requests as one batch. The batch_* methods on Model are the provider-level primitives
both are built on.
One model request to send as part of a batch.
Every aspect of the request can differ between the requests of a batch, as long as the provider accepts them in
one job: Model.batch_encode reports which ones it can combine.
The ID that matches the request to its result. It must be unique within a batch.
Type: str
The messages to send to the model.
Type: list[ModelMessage]
Settings for this request.
Type: ModelSettings | None Default: None
Tools, output schema and other request parameters for this request.
Type: ModelRequestParameters Default: field(default_factory=ModelRequestParameters)
A BatchRequest encoded for one provider’s batch API.
The ID that matches the request to its result within the batch.
Type: str
Everything requests must share to be submitted in one batch, such as the endpoint, credentials and model.
It is stored on the Batch handle, so it never contains secrets.
Type: str
The provider-native request, as JSON.
Type: str
A handle to a submitted batch job.
It holds only plain data, so you can store it, for example with a Pydantic TypeAdapter, and use it from another
process to check on the job, retrieve its results or cancel it.
The provider’s ID for the batch job.
Type: str
The name of the model the batch was submitted to.
Type: str
The name of the provider the batch was submitted to.
Type: str
The group the batch’s requests share.
Used to check that a later call uses a compatible client configuration.
Type: str
The job’s processing state when this handle was last updated.
Type: BatchStatus
When the provider created the job.
Type: datetime
The number of requests in the batch.
Type: int
Provider-specific job details, such as the files a provider writes results to.
Type: dict[str, Any] | None Default: None
The provider’s result for one request of a batch.
Turn it into a ModelResponse with
Model.batch_decode.
The ID of the request this result belongs to.
Type: str
How the provider finished the request.
Type: BatchResultStatus
The provider-native result or error, as JSON, when the request 'succeeded' or 'errored'.
Type: str | None Default: None
The processing state of a batch job. An ended job can hold a mix of per-request outcomes.
Type: TypeAlias Default: Literal['in_progress', 'canceling', 'ended']
How the provider finished one request of a batch.
Type: TypeAlias Default: Literal['succeeded', 'errored', 'expired', 'canceled']