pydantic_ai.realtime.lifecycle
The lifecycle events a RealtimeConnection yields with its
codec events, for implementing a realtime provider: which response, spoken turn, and input each
codec event is about. Most users only need the session-level API in pydantic_ai.realtime.
The lifecycle events a realtime connection yields, for implementing a realtime provider.
The codec vocabulary says what the model said, but not which response it
belongs to, when that response began or ended, or which of the session’s inputs it answers. Only the
connection sees the provider frames that settle those questions, so a
RealtimeConnection states them outright, interleaved with
its codec events in the RealtimeFrames it yields from
frames():
- a response is bracketed by exactly one
ResponseStartedand oneResponseEnded, and its content arrives only in between; ResponseStarted.answersnames the inputs whose reply the response is, which is whatwait_for_reply()waits for;- a spoken user turn starts with
UserTurnStarted, joins the conversation withUserTurnEnded, and gets no more audio or transcript afterUserTurnClosed(which drops one that hadn’t joined); InputAddedmarks where an input joined the provider’s conversation, so history can follow the provider’s order;InputLostandResponseRequestRefusedsettle the inputs no response will ever answer.
An input is identified by its position among every send()
call made on the connection, counting from 0, as
InputRejected.input_index identifies it.
A protocol that identifies its responses, turns, and inputs (as the OpenAI Realtime API does) is best
translated directly. One that identifies none of them, like Gemini Live and OpenAI GPT-Live, can have
InferredLifecycle work them out from the order of what
the connection sends and receives.
What a connection says about the responses, spoken turns, and inputs its codec events are about.
The base of the lifecycle events below, so isinstance(event, LifecycleEvent) tells one apart from a codec
event in a RealtimeFrame.
Bases: LifecycleEvent
A response began: exactly once per response, before any of its content.
The response’s id: the provider’s, or one the connection made up when the provider gives none.
Type: str
The inputs whose request for a response this response satisfies.
Empty for a response the provider started on its own, such as a server-VAD reply to a spoken turn.
Type: tuple[InputId, …] Default: ()
Whether the provider said which inputs answers are ('protocol'), or the connection worked it out
from the requests it had outstanding ('inferred').
Type: AnswersBasis Default: 'protocol'
The spoken user turn a response the provider started on its own is replying to, when known.
Type: str | None Default: None
The response this one carries on from: one the provider ended while saying the exchange isn’t over (Gemini’s
interaction_status=IN_PROGRESS), so whatever waits for that one’s reply waits for this one too.
Type: str | None Default: None
Whether response_id is the provider’s own id, which history records, rather than one the connection made up
for a protocol that gives its responses none.
Type: bool Default: True
Bases: LifecycleEvent
A response ended: exactly once per started response, after all of its content and usage.
The normalized reason the provider finished the response, when it reported one.
Type: FinishReason | None Default: None
The provider’s raw terminal status details, when it reported any.
Type: dict[str, Any] | None Default: None
Works out a connection’s lifecycle events from the order of what it sends and receives.
For a protocol that identifies no responses, spoken turns, or inputs, as Gemini Live and OpenAI GPT-Live
don’t. The connection, in its send, sends each input
inside sending (which reports it, and reports it
failed if sending it raises) and returns what the send settled
(take_pending). It
passes the codec events each provider message makes through
frame_for to get the frame to yield, and reports a
dropped connection (connection_lost). A
connection whose codec events are an async iterable of their own can have
frames do the last two.
What it works out:
- A response starts with its first output (audio, a transcript, a native tool part, a tool call), and ends at
its
ResponseDone; a response that makes tool calls ends with them, or, when they are reported withresponse_usage_follows, at the usage report that follows them. Its id is made up, so it is recorded as no provider’s (ResponseStarted.provider_id=False). - A response answers every input that asked for one (a
strturn, a tool result, aCreateResponse) and wasn’t answered when it started (ResponseStarted.basis='inferred'). - A spoken turn starts with its first input transcript, joins the conversation as the reply to it starts, and
has its whole transcript once that reply ends, unless the transcript is marked final before. A turn the
provider identifies (its transcripts carry an
item_id) is named by that id, and waits for its final transcript. Without input transcription, audio sent since the last reply is a turn of its own, with no transcript, when the model next replies to speech. - An input joins the conversation when it is sent, unless a reply is under way or owed: it then reached the provider while the model was working on that reply, and follows it.
- A dropped connection loses the response under way and every reply still owed.
These are the defaults; a provider that departs from them gets a subclass overriding these extension points:
transcripts_lag_replies(True): whether the provider transcribes the user’s words after the model started answering them.audio_starts_response(True): whether output audio is the model speaking, so a response starts with it.untranscribed_audio_is_a_turn(True): whether audio the model replies to before any transcript of it came is a spoken turn of its own.takes_turn(codec)(False): whether a provider message has the model take the turn, whatever its output shows.
Whether the provider can transcribe the user’s words after the model started answering them, as most do.
A transcript that starts while a reply nobody asked for is under way is then the speech that reply answers,
rather than speech over it. A subclass for a provider that transcribes as the user speaks (and so reports
speech over a reply while the reply is still going) sets it to False.
Type: bool Default: True
Whether output audio is the model speaking, so a response starts with it.
A subclass for a provider whose output is a continuous track, silent between replies, sets it to False, and
says where the model takes the turn with takes_turn instead.
Type: bool Default: True
Whether audio the model replies to before any transcript of it came is a spoken turn of its own.
A subclass for a provider that replies to speech only once it has transcribed it (a cascaded pipeline, say) sets
it to False, so audio no transcript names, like a microphone streaming before a greeting, makes no turn.
Without input transcription (transcribes=False), audio is the only sign of speech, and always makes a turn.
Type: bool Default: True
def __init__(*, transcribes: bool = True) -> None
Track a connection’s lifecycle.
transcribes : bool Default: True
Whether the session has the user’s speech transcribed: if not, audio alone makes a spoken turn. A setting of the session’s (input transcription), not of the provider.
def input_sent(content: RealtimeInput) -> InputId
An input is about to go out: note whether it asks for a reply, and where it joins the conversation.
Call it for every send(), before anything in it can fail, as the session numbers its inputs that way.
Returns the input’s id, for input_failed and input_unanswerable.
InputId
def input_failed(input_id: InputId) -> None
The input never went out (its send raised): forget it.
def input_unanswerable(input_id: InputId) -> None
An input went (or was meant to go) out, but the provider will never reply to it.
def sending(content: RealtimeInput) -> Generator[InputId]
Report an input a connection’s send() is sending, around the sending itself: yields the input’s id.
It is input_sent going in, and
input_failed if the block raises. The
connection then returns take_pending()
(see the example in Implementing a connection).
Generator[InputId]
def take_pending() -> list[LifecycleEvent]
The events the inputs sent since the last frame settled.
Where each joined the conversation, or that it will go unanswered. A connection returns them from
send(), so the session applies them before anything it
reads next, and puts them at the head of a frame of its own (like the one it reports a reconnect in).
frame_for() takes any still waiting into the
frame it makes.
list[LifecycleEvent]
def frame_for(codec: Sequence[RealtimeCodecEvent]) -> RealtimeFrame
The frame to yield for the codec events one provider message makes, with the lifecycle events around them.
RealtimeFrame
def takes_turn(codec: Sequence[RealtimeCodecEvent]) -> bool
Whether the provider message codec makes has the model take the turn, whatever its output shows.
An extension point for a subclass whose connection knows where the model takes the turn better than its
output shows (see audio_starts_response): a response then starts with the message’s audio, or after its
events if it has none, if none is under way. False by default.
@async
def frames(events: AsyncIterable[RealtimeCodecEvent]) -> AsyncIterator[RealtimeFrame]
Frames for a connection’s codec events, one event to a frame, ending with what the end of events loses.
For a connection whose codec events are an async iterable of their own, and end with the connection.
AsyncIterator[RealtimeFrame]
def connection_lost() -> list[LifecycleEvent]
The connection dropped (and is being re-dialed, or is gone): nothing under way on it will finish.
A re-dial never resumes a generation, even when it resumes the session, so the response under way is lost, and so is every reply still owed. A spoken turn the model was about to answer is in the conversation all the same, with what transcript it has.
list[LifecycleEvent]
Bases: LifecycleEvent
The user began a spoken turn (the provider heard speech start, or a commit names a turn it hadn’t).
Bases: LifecycleEvent
A spoken turn joined the provider’s conversation, at this point in its order.
Whether it joined while the user is still saying it (xAI adds its item at speech start): its audio, and with it the turn, then ends with the speech.
Type: bool Default: False
The response it joined ahead of, when the connection learned of the turn only after that response, its reply, had started (Gemini transcribes the user’s words after the model starts answering them).
Type: str | None Default: None
Bases: LifecycleEvent
No more audio or transcript is coming for a spoken turn.
It was cleared, its connection lost, or its transcript is over. One that hadn’t joined the conversation
(UserTurnEnded) is dropped, as it never will; one that had keeps what it has.
Bases: LifecycleEvent
An input joined the provider’s conversation, at this point in its order.
Bases: LifecycleEvent
The provider will never answer these inputs’ request for a response.
The connection dropped the request (a barge-in cut off the response it was waiting behind), or lost it with the connection. The inputs’ content, if any, is unaffected.
Bases: LifecycleEvent
The provider refused the request for a response made for these inputs, so none will answer them.
Bases: LifecycleEvent
What the provider says about one item of a response’s output, to keep on the part that item makes.
Like OpenAI’s phase, which tells an assistant message the model meant as commentary on its way to a tool
call apart from its final answer.
A codec event about a response that has already ended: a repeated terminal, or content trailing it.
The application still receives event, as it always has, but nothing applies it to the conversation: the
response it is about is over. A connection that can’t tell such events apart (most can’t) never yields one.
An input’s position among every send() call made on a connection, as in InputRejected.input_index.
Default: int
Where a response’s answers come from: the provider’s own word, or the connection’s bookkeeping.
Default: Literal['protocol', 'inferred']
How a response ended. 'lost' is a response the connection gave up on: its connection dropped (and the
reconnect did not carry it over) or closed before the provider reported it done.
Default: Literal['completed', 'cancelled', 'failed', 'incomplete', 'lost']
What one message from the provider (or one transition of the connection’s own, like a reconnect) makes.
The events are in order, and the session applies a frame whole before it streams any of it, so whatever reacts to one of its events finds the conversation as far along as the frame takes it.
Default: TypeAliasType('RealtimeFrame', 'list[RealtimeCodecEvent | LifecycleEvent | StaleEvent]')