> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lokutor.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Trace calls with OpenTelemetry

> Send every voice-agent call to Honeycomb, Grafana, New Relic or your own collector as an OpenTelemetry trace: turns, model requests and tools, with timings and token counts. Never what anyone said.

Lokutor can send each of your agent's calls, web and phone, to your observability tool as an [OpenTelemetry](https://opentelemetry.io) trace. A trace shows where a slow turn spent its time (recognition, waiting for the caller to finish, the language model, the first audio), which tools ran and which failed, and how many tokens each request used. Your own spans can sit next to it in the same tool.

It is off until you set an endpoint, and it is per account: every agent on the account sends to the same place.

## What a call looks like

```
lokutor.call                     the call (web or phone)
  lokutor.agent_turn             the agent's opening
    chat                         its model request, tokens
  chat (speculative)             a reply started early
  lokutor.turn                   last word → first audio
    chat                         the reply's request, tokens
  execute_tool book_appointment  a tool call, its outcome
  chat (after_tools)             tokens added after tools
```

Spans are sent in batches every few seconds while the call runs. The `lokutor.call` span itself arrives when the call ends.

## Set it up

1. In the [dashboard](https://app.lokutor.com), open **Settings → Observability**.
2. **Endpoint**: the OTLP endpoint your tool gives you, the value it tells you to put in `OTEL_EXPORTER_OTLP_ENDPOINT`. `/v1/traces` is added unless the address already ends in it.
3. **Headers**: what your tool needs to accept the data, usually an API key. Values are masked once saved.
4. **Save**, then **Send a test trace**. Lokutor exports one span named `lokutor.test` with what you saved and shows your endpoint's answer: accepted, or the error it returned (a wrong key shows as `401`).

Calls that start after you save send their trace. Switch **Export** off to stop sending without losing the settings, or **Remove** to delete them.

| Tool | Endpoint | Header |
| :- | :- | :- |
| Honeycomb | `https://api.honeycomb.io` (EU: `https://api.eu1.honeycomb.io`) | `x-honeycomb-team`: your API key |
| Grafana Cloud | `https://otlp-gateway-prod-<region>.grafana.net/otlp`, from your stack's **OpenTelemetry** tile | `Authorization`: `Basic <base64 of instance-id:token>` |
| New Relic | `https://otlp.nr-data.net` (EU: `https://otlp.eu01.nr-data.net`) | `api-key`: your license key |
| Your own collector | its OTLP/HTTP receiver, e.g. `https://otel.example.com:4318` | whatever it requires |

Any tool that accepts OTLP over HTTP works. Lokutor sends protobuf over **https** only, to a **public** address: an endpoint that resolves to a private, loopback or link-local address is refused, as are redirects to plain http.

## Spans and attributes

Every span carries the resource attributes `service.name` = `lokutor-voice-agent` and `lokutor.account.id`. Names follow OpenTelemetry's [generative AI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) where one exists (`chat`, `execute_tool`, `gen_ai.*`). Lokutor's own are under `lokutor.*`.

### `lokutor.call`

The call, from start to hang-up. Span kind `server`.

| Attribute | |
| :- | :- |
| `lokutor.session.id` | the call's session id, as in your call history and webhooks |
| `lokutor.channel` | `web` or `phone` |
| `lokutor.agent.id` | the agent |
| `lokutor.call.turns` | caller turns answered, set when the call ends |

### `lokutor.turn`

One caller turn, from the last moment the caller was speaking to the first audio of the reply: the wait the caller hears.

| Attribute | Milliseconds of |
| :- | :- |
| `lokutor.turn.index` | (the turn's number in the call, from 1) |
| `lokutor.latency.e2e_ms` | the whole turn: caller's last voiced audio → first audio of the reply |
| `lokutor.latency.hangover_ms` | silence needed before the turn was considered over |
| `lokutor.latency.stt_ms` | speech recognition |
| `lokutor.latency.turn_gate_ms` | deciding the caller had finished (and not just paused) |
| `lokutor.latency.llm_ms` | the language-model request |
| `lokutor.latency.llm_to_tts_ms` | from the reply to the start of speech synthesis |
| `lokutor.latency.tts_first_chunk_ms` | speech synthesis, to the first audio |
| `lokutor.latency.ttfa_ms` | first audio, counted from the end of the silence instead of the last voiced audio |

A stage that did not run, or could not be timed, is left out rather than sent as 0.

### `lokutor.agent_turn`

The agent speaking without the caller having spoken: `lokutor.trigger` is `opening`, `silence_timeout` (a check-in after silence), `farewell`, or `host_note` (after a transfer nobody answered). `lokutor.latency.response_ms` is from the request to its first audio.

### `chat`

A language-model request. Span kind `client`, `gen_ai.operation.name` = `chat`.

| Attribute | |
| :- | :- |
| `lokutor.llm.kind` | `reply` (inside its turn), `after_tools`, or `speculative` |
| `lokutor.llm.outcome` | speculative requests only: `used`, `unused`, `no_response`, `no_transcript`, `failed` or `cancelled` |
| `gen_ai.provider.name` | the provider that answered, when known |
| `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` | tokens, when the provider reported them |

Every token is counted once, so adding `gen_ai.usage.*` up over a trace gives the call's total:

* **reply**: the turn's request.
* **after\_tools**: a turn with tools asks the model again after they run. This span carries what those requests added and has no duration of its own.
* **speculative**: to answer faster, Lokutor may start a reply while the caller is still finishing. When the turn uses it, its tokens are on the turn's `reply` span: the speculative span says `used` and carries none (a reply the turn took over while it was still streaming has no speculative span at all). When it is thrown away (`unused` and the other outcomes), its tokens are here: they were spent all the same.

### `execute_tool {name}`

A tool call: `gen_ai.tool.name`, and `lokutor.tool.outcome` = `ok`, `failed` or `cancelled` (the call ended, or the caller spoke again first). A failed tool also sets the span's status to error. The error's text is not sent: it can contain your system's URLs or data.

## What is not sent

No transcripts, replies, tool arguments, tool results, prompts, phone numbers or error messages: only timings, counts, names you chose (tools, agent id) and outcomes. What was said stays in your call history in the dashboard.

## If your endpoint is down

Tracing never slows a call down. Spans are queued in memory and sent in the background. When your endpoint refuses them or does not answer, each batch is retried for about 20 seconds and then dropped. They are not stored for later. **Send a test trace** shows the endpoint's last answer when something looks missing.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.