Whisper Speech Recognition Accuracy: What the Research Shows
Short answer — Whisper is an open-source speech recognition model from OpenAI. Its research paper reports a WER of 2.7% on clean, read English (LibriSpeech test-clean) for the large-v2 model, and, more importantly, far fewer errors than a comparable model on noisier, more varied recordings. Its accuracy still varies a lot by language, and it has known failure modes such as hallucinated text. Every figure below comes from the paper or the official repository, listed in the Sources section.
Whisper comes up in almost every conversation about speech-to-text: it is free to download, it handles dozens of languages, and many tools build on it. But "Whisper is very accurate" is a vague claim. This article looks at what its authors actually measured, what those numbers mean, and where the model falls short.
A note on scope: this is an article about Whisper as a third-party model. DictoKey is a separate Android app; it uses cloud-based AI speech recognition, and this article makes no claim about which model any particular app runs.
What Is Whisper?
Whisper is an automatic speech recognition (ASR) model released by OpenAI in 2022 and described in the paper Robust Speech Recognition via Large-Scale Weak Supervision (Radford et al.). According to its official repository, it is a Transformer sequence-to-sequence model trained on several speech tasks at once: multilingual transcription, translation of speech into English, and spoken language identification.
Its main idea is scale with "weak supervision". Instead of a small set of carefully transcribed recordings, the authors collected 680,000 hours of audio paired with transcripts found on the internet. Of those hours, 117,000 cover 96 languages other than English, and 125,000 hours are translation data into English. The transcripts are imperfect, but the sheer amount and variety of audio is what the paper credits for the model's robustness.
The code and model weights are released under the MIT License, which is why Whisper appears in so many open-source tools, research projects and commercial products.
How Speech Recognition Accuracy Is Measured
The standard metric is the word error rate, usually shortened to WER. It counts the words that were substituted, deleted or inserted compared with a reference transcript, divided by the number of words in the reference. Lower is better: a WER of 5% means roughly one wrong word in twenty.
Two cautions apply to any WER figure. First, it depends on the test set: clean audiobook narration is far easier than a phone call in a busy room. Second, it depends on text normalization, since punctuation, casing and spelled-out numbers can count as errors or not. The Whisper paper uses its own text normalizer, so its numbers are not directly comparable with figures computed differently.
What the Paper Reports
The most quoted result is on LibriSpeech test-clean, a benchmark of read English audiobooks. There, the large-v2 Whisper model reaches a WER of 2.7%, exactly the same as a wav2vec 2.0 model trained specifically on LibriSpeech.
The interesting part is what happens outside that benchmark. The authors evaluated both models, without any fine-tuning, on a range of other English datasets: podcasts, meetings, phone calls, accented speech and more. Here is an extract of the paper's Table 2 (WER in %, lower is better):
| Dataset | wav2vec 2.0 Large (no LM) | Whisper large-v2 | Relative error reduction |
|---|---|---|---|
| LibriSpeech Clean | 2.7 | 2.7 | 0.0% |
| LibriSpeech Other | 6.2 | 5.2 | 16.1% |
| Common Voice | 29.9 | 9.0 | 69.9% |
| TED-LIUM | 10.5 | 4.0 | 61.9% |
| CHiME-6 | 65.8 | 25.5 | 61.2% |
| Switchboard | 28.3 | 13.8 | 51.2% |
| AMI SDM1 | 67.6 | 36.4 | 46.2% |
| Average (13 datasets, excluding LibriSpeech Clean) | 29.3 | 12.8 | 55.2% |
The takeaway is not that Whisper is "better than humans" or "the most accurate system". It is that two models that look identical on a clean benchmark can behave very differently on messy audio. Averaged over the other datasets, Whisper made 55.2% fewer errors. The paper also notes that its smallest model, with only 39 million parameters, is roughly competitive with the best supervised LibriSpeech model once you leave LibriSpeech.
The authors also compared Whisper with commercial speech recognition services on long recordings, queried in September 2022, and found it performed better than the compared systems on most datasets. They add a fair caveat: some commercial systems may have been trained on those public datasets, so the comparison may not reflect true robustness. Those services have changed since, so treat that comparison as a snapshot.
Accuracy Across Languages
Whisper is often described as multilingual, and it is, but unevenly. The official repository states plainly that performance "varies widely depending on the language", and publishes per-language error rates for large-v3 and large-v2 on the Common Voice 15 and Fleurs datasets.
The paper explains why. Across languages, accuracy is strongly predicted by how much training audio existed for that language: the authors estimate that the error rate halves for every 16-fold increase in training data. Because the dataset came mostly from English-centric parts of the internet, most languages had less than 1,000 hours of training audio. Widely spoken languages with a lot of online content tend to do well; lower-resource languages do noticeably worse.
The later large-v3 model, announced in November 2023, was trained on 1 million hours of weakly labeled audio plus 4 million hours of pseudo-labeled audio. Its announcement reports a 10% to 20% reduction of errors compared with large-v2, for the languages measured on Common Voice 15 and Fleurs.
Noise and Real-World Audio
The paper tested robustness by adding white noise or "pub noise" (ambient sound and indistinct chatter, like a crowded restaurant) to LibriSpeech recordings. Many LibriSpeech-trained models beat Whisper when the audio was barely noisy, but they all degraded quickly and performed worse than Whisper once pub noise reached a signal-to-noise ratio below 10 dB.
In practice, this means a robust model is more forgiving of everyday conditions, but no model is immune. A closer microphone, such as a headset, still makes a bigger difference than any model choice when you dictate somewhere loud.
Model Sizes: Speed vs Accuracy
Whisper comes in several sizes. The official repository lists them as follows, with approximate memory needs and speed relative to the large model:
- tiny: 39 million parameters, about 1 GB of video memory, about 10 times faster
- base: 74 million parameters, about 1 GB, about 7 times faster
- small: 244 million parameters, about 2 GB, about 4 times faster
- medium: 769 million parameters, about 5 GB, about 2 times faster
- large: 1,550 million parameters, about 10 GB, the reference speed
- turbo: 809 million parameters, about 6 GB, about 8 times faster
The turbo model is an optimized version of large-v3 with faster transcription and little loss in accuracy, but it is not trained for translation. Smaller models are easier to run on modest hardware, at the cost of accuracy, so any accuracy figure only makes sense next to the model size it was measured with.
Known Limitations
The paper is candid about what still goes wrong. Its limitations section lists errors that look "decidedly non-human":
- Repeat loops: the model gets stuck repeating the same phrase.
- Missing words at the start or end of an audio segment.
- Hallucination: a transcript entirely unrelated to the actual audio.
- Long recordings: the model processes 30-second chunks, so long-form transcription needs extra decoding strategies.
- Lower-resource languages: accuracy is still poor for many languages with little training data.
Like any speech recognition system, it can also stumble on rare names, brand names and technical vocabulary. None of this cancels its strengths, but it explains why you should read a dictated text before sending it.
What This Means for Voice Typing on Android
For everyday dictation, the lessons from Whisper's research apply to any speech recognition engine: benchmark numbers on clean audio say little about your own voice, accent, language and room. The best test is to dictate a few real messages in the conditions you actually use.
Accuracy is also only part of the experience. Automatic punctuation, how easily you can correct a word, and whether dictation is available where you type often matter just as much. Our guide to voice dictation apps for Android compares the options, and voice typing vs keyboard looks at when speaking beats typing.
DictoKey is one option: an Android voice keyboard and floating dictation bubble that uses cloud-based AI speech recognition (so it needs an internet connection), restores punctuation automatically, and can rewrite or translate what you dictate in 51 languages. It is built first for people for whom typing is painful, slow or unreliable; see our accessibility page.
Try Voice Typing on Your Android Phone
DictoKey: voice keyboard plus floating bubble, automatic punctuation, rewriting and translation in 51 languages, no account.
Download on Google Play Free plan with a daily dictation limit — Android 10+Frequently Asked Questions
Sources
Checked on 2026-10-19.
- Radford et al. — Robust Speech Recognition via Large-Scale Weak Supervision (arXiv:2212.04356) — training data, Table 2, noise robustness, languages, limitations
- openai/whisper on GitHub — architecture, model sizes, turbo, MIT License, per-language performance
- openai/whisper — large-v3 announcement (November 2023) — large-v3 training data and error reduction
Whisper and OpenAI are trademarks of their respective owner. DictoKey is an independent app and is not affiliated with or endorsed by OpenAI.