WebSocket Protocol

Why WebSockets

A voice conversation needs audio flowing in both directions at once, interleaved with control messages, for the length of the call. A WebSocket carries all of it on one connection. This page documents the voice agent protocol (/ws/agent). The plain TTS endpoint (/ws/tts) is much simpler — see Real-time Streaming.

Connection lifecycle

1. Handshake

GET /ws/agent?agent_id=… HTTP/1.1
Upgrade: websocket
Connection: Upgrade
X-API-Key: sk_...

2. Authentication

The credential is checked during the handshake. It can be an Authorization: Bearer header, an X-API-Key header, or the api_key query parameter, and it can be an API key or a short-lived wst_ ticket; browsers, which cannot set WebSocket headers, should pass a ticket as ?api_key=. See Authentication. A bad credential fails the upgrade with 401; other handshake refusals are listed in Voice Chat.

3. Configuration

The SDK sends JSON messages immediately after connecting, in this order:

{ "type": "voice", "data": "F1" }
{ "type": "language", "data": "en" }
{ "type": "tools", "data": [ /* only if any tools are registered */ ] }
{ "type": "visemes", "data": false }
{ "type": "prompt", "data": "You are a helpful assistant." }
{ "type": "rates", "playback": 44100, "input": 16000 }

The exact order doesn’t matter to the server except for rates, which should be sent last: receiving it is what releases the “first turn” gate server-side (see rates below), so sending it before the rest of the configuration risks the first turn running against stale defaults.

4. Streaming

Client → Server: Binary Int16 PCM frames (16 kHz, mono) — or, equivalently, a JSON {"type":"audio","data":"<base64>"} message carrying the same PCM bytes (see Binary vs. JSON audio).
Server → Client: JSON text frames only. Unlike the plain /ws/tts endpoint, /ws/agent never sends raw binary audio frames — agent audio always arrives as a base64-encoded audio JSON message (see below).

Frame types

Binary frames: microphone audio

Direction	Format	Sample Rate	Chunk Size
Client → Server (mic)	Int16 PCM	16 kHz	~320 bytes (20ms)

There is no server → client binary frame on this endpoint — agent audio is always wrapped in a JSON audio message (base64-encoded), not sent raw. This differs from /ws/tts, which does send raw binary audio frames.

Binary vs. JSON audio

The SDK sends microphone audio as raw binary WebSocket frames. The server also accepts the same audio wrapped in a JSON text message — useful for environments where sending arbitrary binary frames is awkward:

{ "type": "audio", "data": "<base64-encoded Int16 PCM>" }

Both forms feed the same audio pipeline; you don’t need to pick one exclusively. The SDKs only ever use the binary form.

Text frames: JSON

Client → Server:

Type	Payload	Purpose
`audio`	`data`: base64 PCM16	Alternative to a raw binary frame — same effect
`voice`	`data`: voice ID string	Set/change voice style
`language`	`data`: language code string	Set/change language
`prompt`	`data`: string	Set/replace the system prompt
`tools`	`data`: array of tool definitions	Register/replace LLM tool definitions
`visemes`	`data`: boolean	Enable/disable lip-sync extraction for subsequent responses
`pitch`	`data`: number (or numeric string)	Set a pitch-shift amount applied to synthesized audio for this session
`rates`	(no payload fields are read by the server)	Client → server “handshake complete” signal — see `rates`
`reset`	(no payload)	Clear the conversation context for this session
`tool_result`	`data`: `{ call_id, result }`	Return the result of a client-executed tool call

Server → Client:

Type	Payload	When
`status`	`data`: see status values	Connection/state changes and message acknowledgments
`server_rates`	`playback`, `input` (sample rates in Hz)	Once on connect, and again in response to a client `rates` message
`transcript`	`role` (`user`\|`agent`), `data`, optional `partial: true`	Speech transcribed (user) or agent text response. `partial: true` marks interim user text; the final line follows without it
`transcript_truncated`	`role: "agent"`, `data`, `full`, `emitted`	The user cut the agent off. `data` is what they heard, ending in `…` (empty if nothing). If `emitted` is true, replace the agent `transcript` whose text was `full`; otherwise add `data` as a new agent line
`transcript_revised`	`role: "user"`, `data`, `previous`	The user’s paused sentence and its continuation were transcribed as one turn. Replace the last user line (`previous`) with `data` and drop any agent line after it
`audio`	`data`: base64 PCM16, `generation`: int	Agent speech, chunked
`visemes`	array of `{ v, c, t }`	Lip-sync metadata, if `visemes` was enabled
`tool_call`	`name`, `arguments` (JSON string), `call_id`	The LLM wants to invoke a tool
`error`	`data`: a structured error object (`code`, `message`, `detail`, `retryable`) or a plain string	Invalid configuration (structured) or a pipeline error (may be a string)
`session_limit`	`data`: `{ seconds, reason }`	Once, right after `connected`, when the session has a time limit (today only the key-less demo on lokutor.com, `reason: "demo"`); the server closes the socket with code 1008 when it runs out. API-key sessions never receive it

`rates`

The SDK sends {"type":"rates","playback":44100,"input":16000} as the last step of its configuration handshake. The server does not currently read playback/input from this message — it treats receipt of any rates message as “the client has finished configuring the session” and uses that to release the gate on the first conversational turn (so the first turn doesn’t start against a stale default prompt/voice/language while the client is still mid-handshake). The server then replies with its own actual rates via server_rates. If you’re implementing the protocol by hand, sending {"type":"rates"} with no fields still works — just send it last.

`status` values

status messages double as both state-machine transitions and lightweight acks for the config messages above:

`data` value	Meaning
`connected`	Handshake complete; also carries a `session_id` field
`listening`	Server is waiting for/receiving user speech
`thinking`	LLM is generating a response; carries a `generation` field
`speaking`	Agent audio is playing
`interrupted`	The user is really interrupting: stop playback and clear queued audio. See barge-in below
`prompt_set` / `voice_set` / `language_set` / `pitch_set` / `rates_set` / `reset`	Acknowledges the matching client config message

There is no reconnecting or disconnected value sent by the server — those are purely client-side SDK states emitted while the WebSocket itself is down (see Reconnection).

Barge-in

See Handling Interruptions for the full picture. Summary for wire-protocol purposes: when the user starts speaking over the agent, the server sends status: listening and stops sending new audio for the response, but the barge-in is only tentative — keep playing what you have. status: interrupted follows once the user has kept talking for 400 ms or the barge-in is confirmed (long enough, not noise, not an echo of the agent’s own voice); stop playback then. A confirmed barge-in also truncates the assistant’s context to what was heard (transcript_truncated). If the tentative barge-in is dismissed as a false alarm, the server sends a status naming what’s true now — speaking if the response is still playing or resumes, thinking if it was still generating one — and the response continues under the same generation. There’s no separate “resumed” type.

Generation numbers

Every response has a generation counter, incremented each time the server starts a brand-new turn (not on every barge-in — a dismissed barge-in keeps the same generation). Discard audio from older generations:

ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);
  if (msg.type === 'status' && msg.data === 'thinking') {
    currentGeneration = msg.generation;
    audioQueue.clear();
  }
  if (msg.type === 'audio' && msg.generation < currentGeneration) {
    return; // Discard stale audio
  }
};

Call sounds

An agent can have a background sound (a room under the whole call) and a typing sound that plays while one of its tools runs. Both are off unless the agent’s owner picks them in the dashboard (agent → Tools → Sounds), from the presets or their own file. When an agent has a background sound, the server mixes it into the audio it sends, the same way on web, API and phone calls. Your client plays it like any other audio, with nothing to change. What is different on the wire:

Audio keeps arriving between replies. While the agent is quiet the server sends short audio messages of the room, about a quarter of a second ahead of playback, so the background never stops. They carry the newest generation your client knows, so the stale-audio check above never drops them.
Clear your queue when a new reply starts (status: thinking with a newer generation) and on status: interrupted, as above. The server assumes you do: the room queued ahead is then discarded and the reply plays at once. A client that never clears its queue still works, but hears each reply up to ~350 ms later.
The agent’s speech arrives with the room already in it. Call recordings keep the voice only.

Reconnection

How the server closes the socket says whether to reconnect:

Close code	Meaning	Reconnect?
1000	The session ended normally (e.g. the agent ended the call, or `nobody has spoken for 5 minutes` when neither side spoke for 5 minutes and the agent said goodbye)	No
1008	A limit was reached; the close reason says which (e.g. the demo’s time limit, or `account balance exhausted` when the account’s balance ran out during a call, a few seconds after the agent’s goodbye)	No
4000–4999	Application-specific end of session	No
1001, 1006, 1011–1014	The server went away or the connection dropped	Yes

The SDK reconnects only in the last case, with exponential backoff: 2s → 4s → 8s → capped at 10s, max 5 attempts. A reconnection is a new server session: send the full configuration again, and reset your generation counter — the new session numbers its responses from 1, so audio compared against the old session’s counter would all be discarded as stale. The SDK emits its own reconnecting / disconnected / reconnected client-side events during this process — these are never sent by the server.

Common pitfalls

binaryType = 'arraybuffer' is required (default is blob)
Don’t send audio before onopen
Handle generation counters for interruption
Use audioContext.currentTime, not Date.now(), for viseme timing
A tentative barge-in that resolves as a false alarm is announced as an ordinary status message (speaking or thinking, not a separate “resumed” type) — see Barge-in.
Keep streaming audio continuously after the user stops talking, not just while they’re speaking. End-of-turn detection is silence-duration-based and only advances when a chunk arrives; a client that stops sending entirely the moment the user goes quiet (rather than continuing to send near-silent mic frames, which any live capture naturally produces) leaves the server with nothing to measure silence against, and the turn never closes. The SDKs already stream continuously and don’t need to do anything special here — this only matters if you’re driving the wire protocol by hand.

Get Started

SDK

API Reference

Resources

WebSocket Protocol

Why WebSockets

Connection lifecycle

1. Handshake

2. Authentication

3. Configuration

4. Streaming

Frame types

Binary frames: microphone audio

Binary vs. JSON audio

Text frames: JSON

`rates`

`status` values

Barge-in

Generation numbers

Call sounds

Reconnection

Common pitfalls

​Why WebSockets

​Connection lifecycle

​1. Handshake

​2. Authentication

​3. Configuration

​4. Streaming

​Frame types

​Binary frames: microphone audio

​Binary vs. JSON audio

​Text frames: JSON

​rates

​status values

​Barge-in

​Generation numbers

​Call sounds

​Reconnection

​Common pitfalls

Why WebSockets

Connection lifecycle

1. Handshake

2. Authentication

3. Configuration

4. Streaming

Frame types

Binary frames: microphone audio

Binary vs. JSON audio

Text frames: JSON

`rates`

`status` values

Barge-in

Generation numbers

Call sounds

Reconnection

Common pitfalls