Audio specs
Agent audio always arrives as a base64-encoded
audio JSON message — this endpoint never sends raw binary audio frames to the client (unlike /ws/tts).
Connection
Agent config
A session with anagent_id runs with that agent’s config, set in the dashboard. Keys documented elsewhere: web_search, google_calendar and the calendar settings, crm and timezone in Tool Calling. This one changes how the agent takes its turn:
Initialization
Send configuration as JSON messages after connecting, in this order:rates last — receiving it is what tells the server your configuration handshake is complete and releases the gate on the first conversational turn. Each of voice/language/prompt/rates is acknowledged with a status message (voice_set, language_set, prompt_set, rates_set).
Client to server
Send microphone audio as binary Int16 PCM frames (16 kHz, mono, ~20ms chunks). You can also send the same audio as{"type":"audio","data":"<base64>"} if binary frames aren’t convenient in your environment.
Other client → server messages: pitch (adjust pitch-shift), reset (clear conversation context), tool_result (return a client-executed tool’s result — see Tool Calling).
Server to client
Barge-in
When the user speaks while the agent is talking, the server immediately sendsstatus: interrupted (tentatively — audio for the current response stops arriving right away). It then either confirms the interrupt (a second status: interrupted, plus a truncated context) using a fast on-device classifier (GateTurn) or, failing that, the existing speech-based checks — or dismisses it as a false alarm, in which case no further message is sent and audio for the original response simply resumes under the same generation. See Handling Interruptions for the full mechanism.
An agent reply is sent as a transcript as soon as it is generated, which is before it has finished playing. When the user interrupts, a transcript_truncated follows with what they actually heard, and the agent’s own memory of the conversation is corrected the same way. If the user was only pausing and the agent had said no more than a word or two, the user’s two fragments become one turn (transcript_revised) and the agent answers the whole sentence.
A reply the user never heard any of — they started speaking again before it began to play — is not kept. It is either never sent as a transcript, or, if it already was, withdrawn with a transcript_truncated whose data is empty and emitted true. The agent forgets it too, and answers what the user goes on to say.
Silence
If the user says nothing for 10 seconds after the agent finishes speaking, the agent speaks once, unprompted: it asks whether they are still there or need anything else — or, if their last turn went unanswered, answers it. It arrives like any other reply (transcript, then audio under a new generation), at most once per silence; the next time the user speaks re-arms it. It never acts on the user’s behalf: no tool is called while they are silent, even if the agent’s last line offered to do something.
Related Pages
- WebSocket Protocol — full message reference
- Handling Interruptions — barge-in mechanism in detail
- Tool Calling —
tool_call/tool_resultflow and built-in tools