Optional_Live OTel context of the CURRENT voice.turn, published by
defaultVoiceCall for background-receive-loop adapters
(Pipecat/Twilio) to parent their detached-callback recv spans under the
turn (#774 — the reusable pattern Twilio PR5 inherits). undefined between
turns, so a callback firing outside a turn skips its span rather than
parenting under a closed turn. Internal (underscore) — not a public API.
SET when the AGENT deliberately ended the call (e.g. an ElevenLabs hosted
agent invoking the end_call system tool), as opposed to the transport
dropping. A scripted turn arriving after this concludes the conversation
instead of failing the run — the agent behaved as designed. Assertions and
judges can read it to reason about WHO ended the call.
OptionalagentSet when the adapter has emitted its first agent audio chunk for the
current turn — gates timing-based barge-in. Concrete adapters expose
this so scenario.interrupt can wait for real speech before
firing the interruption. Optional: adapters without server-VAD-style
interrupt sequencing can leave it undefined.
ReadonlycapabilitiesDeclaration of what this adapter can and cannot do. Concrete subclasses MUST publish a non-default value; the base instance defaults to "nothing supported" so capability-gated steps fail safely when an adapter forgets to declare.
ReadonlyinstructionsMost recent finalized agent transcript (post audio_transcript.done).
Most recent user-side transcript from the Whisper input pipeline.
ReadonlymodelOptionalnameHard cap on a single agent turn's audio. Prevents runaway loops if a transport never signals end-of-stream. 30s = a long sentence.
Tail silence: once the first agent chunk arrives, keep draining receiveAudio until no chunk shows up within this many seconds — that's how we detect the agent finished talking.
Seconds to wait for agent audio after sending user audio: the STT + LLM +
TTS budget for one agent turn. Kept identical to Python's
VoiceAgentAdapter.response_timeout so the same scenario passes or fails
the same way in both SDKs.
Raise it for an agent that runs a tool call or a retrieval step before it speaks:
const agent = elevenLabsAgent({ agentId, apiKey });
agent.responseTimeout = 180; // wait up to 3 minutes
OptionalstreamingIncremental transcript text emitted while the agent speaks. Populated
by adapters that advertise capabilities.streamingTranscripts. Read
by scenario.interrupt when afterWords: N is set.
ReadonlytoolsBounded grace-wait (seconds) for the agent turn's transcript AFTER audio
drains (#734). Audio silence closes the turn (responseTailSilence), but a
live voice agent (hosted ElevenLabs) delivers the turn's text on a SEPARATE
socket event (agent_response → lastAgentTranscript). When that event
lands after the audio-silence boundary, snapshotting lastAgentTranscript
at drain-close reads null and the turn reaches the text-only simulator as
a bare [audio message] — the simulator then fabricates.
The default call() flow (defaultVoiceCall) polls this field up to
this ceiling for a pending transcript before reading it. It short-circuits
the INSTANT lastAgentTranscript is already set (zero added latency on the
happy path — the common case where the transcript won the race) and only
elapses when the transcript genuinely never arrives, so a real ElevenLabs
drop still terminates the turn. Set to 0 to disable the wait.
ReadonlyvoiceSurface realtime tool calls alongside the spoken audio turn (#630).
The base call() (defaultVoiceCall) returns a single assistant audio
message and does all the recording bookkeeping. We keep that intact and,
when the agent called any tools this turn, append ONE extra role:"tool"
message carrying every call as AI-SDK tool-result parts — the shape
state.hasToolCall / state.lastToolCall consume (AC4).
Returns:
[audioMessage, toolMessage] when ≥1 tool was called (AC4/AC10).
convertAgentReturnTypesToMessages passes a list through verbatim into
the run's messages.Per-turn tool state is reset HERE (turn start) so tool calls never leak
across turns; the function-call events for THIS turn are consumed inside
super.call()'s drain and finalized onto _completedToolCalls.
Open the Realtime WebSocket and send the initial session.update.
Close the WebSocket if open.
Send response.cancel — the OpenAI Realtime API's first-class
interrupt. The model stops generating audio and text immediately. No
timing race against VAD: deterministic stop, then the next user turn
flows normally through sendAudio + receiveAudio.
Whether the Realtime WebSocket is open (Gap #11).
Commit any pending audio, request a response, and return the first audio chunk the model produces.
Loops over incoming events until a response.output_audio.delta
event arrives, then returns decoded PCM16. Transcript events update
lastUserTranscript / lastAgentTranscript. An error event throws.
GA event names are response.output_audio[_transcript].{delta,done}
(the Beta response.audio[_transcript].* names are dead). We accept
both so back-port to a Beta endpoint stays trivial; production hits
the GA path.
Append a PCM16 audio chunk to the model's input audio buffer.
Only emits input_audio_buffer.append — commit + response are deferred
to the next receiveAudio call. The executor may call sendAudio many
times for a single user turn (TTS streams audio as chunks); committing
per-chunk would confuse the server with sub-second turn boundaries.
Transmit DTMF tones to the telephony peer. Adapters that advertise
capabilities.dtmf MUST implement this; the default raises
UnsupportedCapabilityError so an adapter that forgot to ship
sendDtmf while claiming the capability fails loudly instead of
silently routing through a PCM fallback.
Inject scripted text into the realtime session as a user message.
Used when this adapter is the user simulator (role=USER): scripted
user("text") steps route through here instead of spawning TTS. The
model synthesizes the text into spoken audio with natural prosody,
which is then delivered via receiveAudio.
Per §7.2, OpenAI Realtime cannot populate assistant audio messages retroactively; the downstream transcript reflects what the model actually emitted, not what was scripted.
Speak a scripted user line AND drain the spoken audio the realtime model
synthesizes for it — the bridge that lets a realtime USER feed a SEPARATE
agent-under-test (e.g. hosted ElevenLabs) through scenario.run() (#705).
Why NOT sendText here: sendText adds the line as a role:"user"
conversation item and calls a bare response.create. Per the OpenAI
Realtime semantics, the model then treats the scripted line as USER input
and GENERATES an assistant reply to it — so a scripted user QUESTION ("what
are your support hours?") comes back ANSWERED ("our hours are 8am–8pm"),
which makes the realtime user sound like the agent and the conversation
incoherent (observed live, #705). For a user SIMULATOR we instead drive a
VERBATIM render: response.create with an instructions override telling
the model to SPEAK the line as the customer, not answer it. The spoken
audio then arrives as response.output_audio.delta frames.
This method loops receiveAudio until the model stops speaking (the
realtime adapter THROWS a timeout once the audio deltas stop — that is the
natural end-of-turn signal, identical to the loop in the adapter-level demo
test), merges the PCM16, and returns ONE AudioChunk carrying:
transcript = the model's own spoken transcript
(lastAgentTranscript, populated from response.output_audio_transcript.done),
falling back to the scripted text if the transcript never arrived.The transcript is load-bearing for the hosted EL transport: its default
turnCommitMode:"text" commits the user turn via a user_message event
built from the AudioChunk transcript (it does not ingest raw user audio).
So the realtime user genuinely speaks (audio out) AND the agent-under-test
receives a real, committed turn (transcript in) — realtime → realtime over
the scenario API, no TTS.
Serial by construction: the turn is fully drained before returning, so the
realtime session never has two concurrent response.creates in flight —
sidestepping the active-response race (scenario#657) on the scripted path.
CONTRACT (realtime user): user("text") is best-effort / INTENT, not exact.
The realtime model voices the line naturally and MAY rephrase it (it treats
"say it verbatim" as framing — live #705: the scripted "...account balance"
line came out reworded as "...check my current account balance"). This is
accepted as the speech-native behaviour (realtime over TTS); for EXACT,
word-for-word scripted lines, use a text or TTS user instead, where
user("text") is delivered verbatim. See decision
2026-06-30-realtime-user-into-proceed.md (#38).
The scripted user line to voice (best-effort verbatim — the model may rephrase; see CONTRACT above).
Per-frame idle timeout while draining (seconds). The loop ends on the first frame that doesn't arrive within this window. Defaults to 15 (matches the demo test's drain window).
The merged spoken-audio chunk (empty data if the model produced
no audio), with transcript set as described above.
Hide the API key when this object lands in error messages or logs.
Exercise OpenAI's Realtime API as either the agent under test (
role=AGENT, default) or as the voice-enabled user simulator (role=USER, per §7.2 L1164-1171).When
role=USER, scripteduser("text")steps route text through the realtime session's text-input channel rather than triggering TTS.Transcript observability:
lastUserTranscript— set fromconversation.item.input_audio_transcription.completedlastAgentTranscript— accumulated fromresponse.audio_transcript.delta/ reset on done