The lowest errors on every benchmark.
Scored on public meeting recordings against pyannote, AssemblyAI and Deepgram. Lower is better.
26%
fewer errors than the next best system, on average across the public meeting benchmarks
- NOTSOFAR office meetings
- 43%
- fewer errors than AssemblyAI and pyannote
- Deepgram Nova-3
- 54–73%
- fewer errors, depending on the benchmark
- AMI headset meetings
- Best
- published result, ahead of Gemini and VibeVoice
Against the leading commercial APIs
Wrong words plus words given to the wrong speaker, each system through its own batch API on the same audio.
NOTSOFAR, AMI one mic and DiPCo
26%lower than the next best
Calico23.3
Precision-331.5
Universal-3.5 Pro31.6
Nova-360.3
NOTSOFAR office meetings
43%lower than the next best
Calico17.7
Precision-333.0
Universal-3.5 Pro31.2
Nova-365.7
AMI meetings
One distant microphone
10%lower than the next best
Calico19.1
Precision-321.2
Universal-3.5 Pro25.5
Nova-343.1
AMI meetings
Headset mix
33%lower than the next best
Calico11.8
Precision-317.6
Universal-3.5 Pro20.8
Nova-336.8
DiPCo dinner parties
Far field
13%lower than the next best
Calico33.2
Precision-340.3
Universal-3.5 Pro38.1
Nova-372.1
Against published research systems
The standard AMI test meetings: tcpWER (it also checks when each word was said) / cpWER.
| System | One distant microphone | Headset mix |
|---|---|---|
| Calico | 19.5 / 18.8 | 11.7 / 11.4 |
| DM-ASR 1.7B (2026) | 22.1 / 21.3 | 16.9 / 16.4 |
| VibeVoice-ASR 7B | 29.8 / 28.8 | 20.8 / 20.4 |
| Gemini 2.5 Pro | 41.4 / 34.8 | 38.4 / 29.6 |
| Gemini 3.0 Pro | 64.9 / 26.9 | 63.7 / 22.3 |
| DiariZen + Whisper large-v3 | 49.1 / 43.9 | 34.0 / 32.3 |
How this was measured
- Every system got the same audio through its batch API, with default settings and speaker labels on. Tests were run in October 2026.
- The comparison with commercial APIs is scored with AssemblyAI's open-source benchmark method: the Whisper text normalizer and meeteval cpWER, averaged over recordings.
- The comparison with research systems uses BUT's EMMA leaderboard references and scoring: meeteval tcpWER with a 5-second timing tolerance and the CHiME-8 text normalizer. Research numbers are as published.
- Part of AMI was in the training data of most of the research systems.