> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lokutor.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmarks

> Measured accuracy of Lokutor Conv 2.0 (speech to text) and Versa 2.0 (text to speech) on public test sets, with the method, so anyone can reproduce it.

Every number here is a measurement we ran ourselves through the public API (`api.lokutor.com`), on public test sets, on 3 and 4 November 2026. Raw per-utterance outputs and the scripts are kept; ask us for them. We show the weak results next to the strong ones.

## Speech to text: Conv 2.0

Word error rate (WER, lower is better) with the [Open ASR Leaderboard](https://github.com/huggingface/open_asr_leaderboard) code and normaliser, through `POST /v1/speech-to-text` (ElevenLabs-compatible). Our submission to the leaderboard is [pull request #267](https://github.com/huggingface/open_asr_leaderboard/pull/267).

**English, short-form**

| Test set | WER % |
| :- | -: |
| LibriSpeech test-clean | 2.84 |
| SPGISpeech | 3.99 |
| VoxPopuli | 4.11 |
| Monsoon (en-IN) | 5.83 |
| URGENT 2024 (clean) | 6.33 |
| LibriSpeech test-other | 6.69 |
| GigaSpeech | 8.97 |
| AMI | 9.48 |
| Earnings22 | 10.81 |
| URGENT 2024 (noisy) | 15.34 |

The Open ASR average over its nine standard sets (URGENT clean excluded) is **7.56%**; over the seven classic sets (AMI, Earnings22, GigaSpeech, both LibriSpeech sets, SPGISpeech, VoxPopuli) it is 6.70%. Throughput as the leaderboard defines RTFx: 13.6 on average, measured per request over the internet with 8 requests in flight, so it is a per-request figure, not server capacity.

**Other languages**

| Language | FLEURS | Common Voice (5,000-clip sample) | MLS |
| :- | -: | -: | -: |
| Spanish | 6.14 | 8.76 | 5.65 |
| Italian | 6.94 | 8.84 | 9.47 |
| German | 9.00 | 9.11 | n/a |
| Portuguese | 9.22 | 4.73 | 12.00 |
| French | 10.45 | 11.73 | 7.23 |

**Basque and Spanish** ([HiTZ test sets](https://huggingface.co/datasets/HiTZ/benchmark_eseu_testsets), Whisper basic normaliser, so comparable with HiTZ's model cards): Basque **10.1%** (4.4 h; Common Voice 8.3, parliament 6.4, OpenSLR 21.7) and Spanish **9.0%** (7.5 h; parliament 5.0, MLS 6.5, VoxPopuli 10.0, Common Voice 11.4, OpenSLR 13.5).

Details that matter: Common Voice rows are random 5,000-clip samples (seed 42) of the full test sets; three Gigaspeech and 34 AMI predictions came back empty and are counted as deletions; the model writes numerals where some references spell numbers out.

## Text to speech: Versa 2.0

**Seed-TTS-Eval, English** (1,088 Common Voice utterances, WER only, Whisper-large-v3 as the recogniser): **1.82%** with voice F1 and **1.90%** with voice M1. We report WER only: the benchmark's speaker-similarity score compares the output to a prompt speaker, and our stock voices are not conditioned on that clip, so the number would not mean anything. This is also not the benchmark's zero-shot cloning task. The scoring path was re-implemented for English (the official script needs CUDA); normalisation follows the repository's `run_wer.py`.

**Speed and capacity** are measured separately and published in the [models page](https://lokutor.com/models): about 90 ms to first audio under load.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.