pydantic_ai.realtime.lifecycle
The lifecycle events a RealtimeConnection yields with its
codec events, for implementing a realtime provider: which response, spoken turn, and input each
codec event is about. Most users only need the session-level API in pydantic_ai.realtime.
The lifecycle events a realtime connection yields, for implementing a realtime provider.
The codec vocabulary says what the model said, but not which response it
belongs to, when that response began or ended, or which of the session’s inputs it answers. Only the
connection sees the provider frames that settle those questions, so a
RealtimeConnection states them outright, interleaved with
its codec events in the RealtimeFrames it yields from
frames():
- a response is bracketed by exactly one
ResponseStartedand oneResponseEnded, and its content arrives only in between; ResponseStarted.answersnames the inputs whose reply the response is, which is whatwait_for_reply()waits for, andAnswersRevisedcorrects them as the connection learns better:claimedwhile the response is under way,releaseduntil its reply (tool rounds included) is over;- a spoken user turn starts with
UserTurnStarted, joins the conversation withUserTurnEnded, and gets no more audio or transcript afterUserTurnClosed(which drops one that hadn’t joined); InputAddedmarks where an input joined the provider’s conversation, so history can follow the provider’s order;InputLostandResponseRequestRefusedsettle the inputs no response will ever answer, andToolCallsAbandonedthe tool calls whose results none will;OutputItemDetailskeeps what the provider says about an output item on the part it makes, andOutputItemCutrecords that the listener heard only part of it, even after its response ended.
An input is identified by its position among every send()
call made on the connection, counting from 0, as
InputRejected.input_index identifies it.
A protocol that identifies its responses, turns, and inputs (as the OpenAI Realtime API does) is best
translated directly. One that identifies none of them, like Gemini Live and OpenAI GPT-Live, can have
InferredLifecycle work them out from the order of what
the connection sends and receives.
What a connection says about the responses, spoken turns, and inputs its codec events are about.
The base of the lifecycle events below, so isinstance(event, LifecycleEvent) tells one apart from a codec
event in a RealtimeFrame.
Bases: LifecycleEvent
A response began: exactly once per response, before any of its content.
The response’s id: the provider’s, or one the connection made up when the provider gives none.
Type: str
The inputs whose request for a response this response satisfies.
Empty for a response the provider started on its own, such as a server-VAD reply to a spoken turn.
Type: tuple[InputId, …] Default: ()
Whether the provider said which inputs answers are ('protocol'), or the connection worked it out
from the requests it had outstanding ('inferred').
Type: AnswersBasis Default: 'protocol'
The spoken user turn a response the provider started on its own is replying to, when known.
Type: str | None Default: None
The response this one carries on from: one the provider ended while saying the exchange isn’t over (Gemini’s
interaction_status=IN_PROGRESS), so whatever waits for that one’s reply waits for this one too.
Type: str | None Default: None
Whether response_id is the provider’s own id, which history records, rather than one the connection made up
for a protocol that gives its responses none.
Type: bool Default: True
Bases: LifecycleEvent
Which inputs a response answers turned out different from what its ResponseStarted said.
A connection that works its responses’ answers out (basis='inferred') can learn better as the response goes
on: GPT-Live answers text sent mid-reply in the speech already under way, and Gemini cuts off a reply that an
input reaches mid-generation, leaving that input to the next one. claimed is valid while the response is under
way, released until its reply (tool rounds included) is over.
It moves no message: history keeps every input where it joined the conversation.
Inputs still owed a reply that this response answers after all. Only while it is under way (before its
ResponseEnded).
Type: tuple[InputId, …] Default: ()
Inputs this response was said to answer that it doesn’t: a later response answers them, naming them in its
ResponseStarted.answers (or in a claimed of its own).
Before its ResponseEnded, or after it while its reply goes on: in the answer to its tool calls, say, which the
provider can cancel when an input cuts the turn off.
Type: tuple[InputId, …] Default: ()
Works out a connection’s lifecycle events from the order of what it sends and receives.
For a protocol that identifies no responses, spoken turns, or inputs, as Gemini Live and OpenAI GPT-Live
don’t. The connection, in its send, sends each input
inside sending (which reports it, and reports it
failed if sending it raises) and returns what the send settled
(take_pending). It
passes the codec events each provider message makes through
frame_for to get the frame to yield, and reports a
dropped connection (connection_lost). A
connection whose codec events are an async iterable of their own can have
frames do the last two.
What it works out:
- A response starts with its first output (audio, a transcript, a native tool part, a tool call), and ends at
its
ResponseDone; a response that makes tool calls ends with them, or, when they are reported withresponse_usage_follows, at the usage report that follows them. Its id is made up, so it is recorded as no provider’s (ResponseStarted.provider_id=False). - A response whose calls are reported with
response_continuesgoes on past them, as the model keeps talking, until itsResponseDone, until the result of one of them goes out (a result directly follows the response that made its call), or until a spoken turn joins the conversation, which follows it too. What the model says after that, before theResponseDonethat ends its turn, is a response of its own. If the model was mid-sentence when the result went out, that is the rest of the sentence, which answers nothing; otherwise it answers the result. Cut off while one of those calls still runs, the response endscompleted: its tool round goes on. - A response answers every input that asked for one (a
strturn, a tool result, aCreateResponse) and wasn’t answered when it started (ResponseStarted.basis='inferred'). - A spoken turn starts with its first input transcript, joins the conversation as the reply to it starts, and
has its whole transcript once that reply ends, unless the transcript is marked final before. A turn the
provider identifies (its transcripts carry an
item_id) is named by that id, and waits for its final transcript. Without input transcription, audio sent since the last reply is a turn of its own, with no transcript, when the model next replies to speech. - An input joins the conversation when it is sent, unless a reply is under way or owed: it then reached the provider while the model was working on that reply, and follows it.
- A dropped connection loses the response under way and every reply still owed.
These are the defaults; a provider that departs from them gets a subclass overriding these extension points:
transcripts_lag_replies(True): whether the provider transcribes the user’s words after the model started answering them.audio_starts_response(True): whether output audio is the model speaking, so a response starts with it.untranscribed_audio_is_a_turn(True): whether audio the model replies to before any transcript of it came is a spoken turn of its own.inputs_cut_off_replies(False): whether an input that reaches the provider mid-generation cuts the reply off, leaving it to the next.takes_turn(codec)(False): whether a provider message has the model take the turn, whatever its output shows.
Whether the provider can transcribe the user’s words after the model started answering them, as most do.
A transcript that starts while a reply nobody asked for is under way is then the speech that reply answers,
rather than speech over it. A subclass for a provider that transcribes as the user speaks (and so reports
speech over a reply while the reply is still going) sets it to False.
Type: bool Default: True
Whether output audio is the model speaking, so a response starts with it.
A subclass for a provider whose output is a continuous track, silent between replies, sets it to False, and
says where the model takes the turn with takes_turn instead.
Type: bool Default: True
Whether an input that reaches the provider while it generates a reply cuts that reply off.
The next reply then answers the input, so a reply cut off (ResponseStarted through a cancelled ResponseEnded)
answers only the oldest input it was taken to answer, as that one started it, or none if it said nothing; and a
reply whose tool calls the provider cancels answers only the oldest too. That is, unless the provider heard the
user start speaking after those inputs went out (speech_heard()):
then the user may have cut the reply off, as ever. The rest are released
(AnswersRevised.released) to the next reply. A
subclass for a provider that cuts its reply off this way (Gemini Live’s 3.x models do) sets it to True.
Type: bool Default: False
Whether audio the model replies to before any transcript of it came is a spoken turn of its own.
A subclass for a provider that replies to speech only once it has transcribed it (a cascaded pipeline, say) sets
it to False, so audio no transcript names, like a microphone streaming before a greeting, makes no turn.
Without input transcription (transcribes=False), audio is the only sign of speech, and always makes a turn.
Type: bool Default: True
def __init__(*, transcribes: bool = True) -> None
Track a connection’s lifecycle.
transcribes : bool Default: True
Whether the session has the user’s speech transcribed: if not, audio alone makes a spoken turn. A setting of the session’s (input transcription), not of the provider.
def input_sent(content: RealtimeInput) -> InputId
An input is about to go out: note whether it asks for a reply, and where it joins the conversation.
Call it for every send(), before anything in it can fail, as the session numbers its inputs that way.
Returns the input’s id, for input_failed and input_unanswerable.
InputId
def input_failed(input_id: InputId) -> None
The input never went out (its send raised): forget it.
def input_unanswerable(input_id: InputId) -> None
An input went (or was meant to go) out, but the provider will never reply to it.
def sending(content: RealtimeInput) -> Generator[InputId]
Report an input a connection’s send() is sending, around the sending itself: yields the input’s id.
It is input_sent going in, and
input_failed if the block raises. The
connection then returns take_pending()
(see the example in Implementing a connection).
Generator[InputId]
def take_pending() -> list[LifecycleEvent]
The events the inputs sent since the last frame settled.
Where each joined the conversation, or that it will go unanswered. A connection returns them from
send(), so the session applies them before anything it
reads next, and puts them at the head of a frame of its own (like the one it reports a reconnect in).
frame_for() takes any still waiting into the
frame it makes.
list[LifecycleEvent]
def speech_heard(*, after: InputId | None = None) -> None
The provider heard the user start speaking, which may cut a reply off (see inputs_cut_off_replies).
A RealtimeInputSpeechStartEvent says so; a connection whose provider reports speech some other way (Gemini
Live’s voice_activity) calls this, or every reply cut off is taken for one an input cut off. after is the id
of the last input sent before the user started speaking, when the provider says where in the audio that was:
-1 if they started before any input, and by default every input sent so far, as the report may come late.
def frame_for(codec: Sequence[RealtimeCodecEvent]) -> RealtimeFrame
The frame to yield for the codec events one provider message makes, with the lifecycle events around them.
RealtimeFrame
def takes_turn(codec: Sequence[RealtimeCodecEvent]) -> bool
Whether the provider message codec makes has the model take the turn, whatever its output shows.
An extension point for a subclass whose connection knows where the model takes the turn better than its
output shows (see audio_starts_response): a response then starts with the message’s audio, or after its
events if it has none, if none is under way. False by default.
@async
def frames(events: AsyncIterable[RealtimeCodecEvent]) -> AsyncIterator[RealtimeFrame]
Frames for a connection’s codec events, one event to a frame, ending with what the end of events loses.
For a connection whose codec events are an async iterable of their own, and end with the connection.
AsyncIterator[RealtimeFrame]
def connection_lost(*, asked_again: Collection[InputId] = ()) -> list[LifecycleEvent]
The connection dropped (and is being re-dialed, or is gone): nothing under way on it will finish.
A re-dial never resumes a generation, even when it resumes the session, so the response under way is
lost, and so is every reply still owed, except those in asked_again: inputs whose reply the re-dial asks
for again, which the next response answers (or which input_unanswerable gives up on, if the re-dial
fails). A spoken turn the model was about to answer is in the conversation all the same, with what
transcript it has.
list[LifecycleEvent]
Bases: LifecycleEvent
A response ended: exactly once per started response, after all of its content and usage.
The normalized reason the provider finished the response, when it reported one.
Type: FinishReason | None Default: None
The provider’s raw terminal status details, when it reported any.
Type: dict[str, Any] | None Default: None
Bases: LifecycleEvent
The user began a spoken turn (the provider heard speech start, or a commit names a turn it hadn’t).
Bases: LifecycleEvent
A spoken turn joined the provider’s conversation, at this point in its order.
Whether it joined while the user is still saying it (xAI adds its item at speech start): its audio, and with it the turn, then ends with the speech.
Type: bool Default: False
The response it joined ahead of, when the connection learned of the turn only after that response, its reply, had started (Gemini transcribes the user’s words after the model starts answering them).
Type: str | None Default: None
Bases: LifecycleEvent
No more audio or transcript is coming for a spoken turn.
It was cleared, its connection lost, or its transcript is over. One that hadn’t joined the conversation
(UserTurnEnded) is dropped, as it never will; one that had keeps what it has.
Bases: LifecycleEvent
An input joined the provider’s conversation, at this point in its order.
Bases: LifecycleEvent
The provider will never answer these inputs’ request for a response.
The connection dropped the request (a barge-in cut off the response it was waiting behind), or lost it with the connection. The inputs’ content, if any, is unaffected.
Bases: LifecycleEvent
The provider refused the request for a response made for these inputs, so none will answer them.
Bases: LifecycleEvent
The provider gave up on these tool calls: no response will answer their results.
Eleven Realtime gives up on a finished response’s calls once it starts a reply to the user before their results are
in, and GPT-Live on the calls its delegated backend asked for when that backend fails. The calls still run, and each
result is recorded in history (and still sent, where the provider takes it), but nothing waits for a reply to it:
the exchange is over once nothing else is owed. Unlike a
ToolCallCancelled, it doesn’t stop them.
The calls, as their ToolCalls named them.
Bases: LifecycleEvent
What the provider says about one item of a response’s output, to keep on the part that item makes.
Like OpenAI’s phase, which tells an assistant message the model meant as commentary on its way to a tool
call apart from its final answer.
Bases: LifecycleEvent
The provider says the listener heard only part of one spoken output item, before or after its response ended.
Generation outruns playback, so the reply the user cuts off has usually ended already. Its message in history is
marked interrupted, and swapped for a copy in the same position, as one the session cuts off itself on a barge-in
is: a snapshot of history already handed out doesn’t change. Like ElevenLabs’ agent_response_correction.
The output item, as the codec events streaming it named it (OutputTranscript.item_id).
Type: str
What the listener heard of the item’s transcript, which history then records instead, keeping the provider’s
original in the part’s provider_details['original_transcript']: '' for an item they heard none of. None
leaves the transcript as it is.
Type: str | None Default: None
How much of the item’s audio the listener heard, in milliseconds, if known.
Type: int | None Default: None
A codec event about a response that has already ended: a repeated terminal, or content trailing it.
The application still receives event, as it always has, but nothing applies it to the conversation: the
response it is about is over. A connection that can’t tell such events apart (most can’t) never yields one.
An input’s position among every send() call made on a connection, as in InputRejected.input_index.
Default: int
Where a response’s answers come from: the provider’s own word, or the connection’s bookkeeping.
Default: Literal['protocol', 'inferred']
How a response ended. 'lost' is a response the connection gave up on: its connection dropped (and the
reconnect did not carry it over) or closed before the provider reported it done.
Default: Literal['completed', 'cancelled', 'failed', 'incomplete', 'lost']
What one message from the provider (or one transition of the connection’s own, like a reconnect) makes.
The events are in order, and the session applies a frame whole before it streams any of it, so whatever reacts to one of its events finds the conversation as far along as the frame takes it.
Default: TypeAliasType('RealtimeFrame', 'list[RealtimeCodecEvent | LifecycleEvent | StaleEvent]')