Whisper Speech Recognition Accuracy: What the Research Shows

By DictoKey Team · Published · Updated — 7 min read

Short answer — Whisper is an open-source speech recognition model from OpenAI. Its research paper reports a WER of 2.7% on clean, read English (LibriSpeech test-clean) for the large-v2 model, and, more importantly, far fewer errors than a comparable model on noisier, more varied recordings. Its accuracy still varies a lot by language, and it has known failure modes such as hallucinated text. Every figure below comes from the paper or the official repository, listed in the Sources section.

Whisper comes up in almost every conversation about speech-to-text: it is free to download, it handles dozens of languages, and many tools build on it. But "Whisper is very accurate" is a vague claim. This article looks at what its authors actually measured, what those numbers mean, and where the model falls short.

A note on scope: this is an article about Whisper as a third-party model. DictoKey is a separate Android app; it uses cloud-based AI speech recognition, and this article makes no claim about which model any particular app runs.

680K
Hours of training audio
96
Languages besides English in the training data
55.2%
Fewer errors on average than a comparable model, on 13 other test sets
MIT
License for code and model weights

What Is Whisper?

Whisper is an automatic speech recognition (ASR) model released by OpenAI in 2022 and described in the paper Robust Speech Recognition via Large-Scale Weak Supervision (Radford et al.). According to its official repository, it is a Transformer sequence-to-sequence model trained on several speech tasks at once: multilingual transcription, translation of speech into English, and spoken language identification.

Its main idea is scale with "weak supervision". Instead of a small set of carefully transcribed recordings, the authors collected 680,000 hours of audio paired with transcripts found on the internet. Of those hours, 117,000 cover 96 languages other than English, and 125,000 hours are translation data into English. The transcripts are imperfect, but the sheer amount and variety of audio is what the paper credits for the model's robustness.

The code and model weights are released under the MIT License, which is why Whisper appears in so many open-source tools, research projects and commercial products.

How Speech Recognition Accuracy Is Measured

The standard metric is the word error rate, usually shortened to WER. It counts the words that were substituted, deleted or inserted compared with a reference transcript, divided by the number of words in the reference. Lower is better: a WER of 5% means roughly one wrong word in twenty.

Two cautions apply to any WER figure. First, it depends on the test set: clean audiobook narration is far easier than a phone call in a busy room. Second, it depends on text normalization, since punctuation, casing and spelled-out numbers can count as errors or not. The Whisper paper uses its own text normalizer, so its numbers are not directly comparable with figures computed differently.

What the Paper Reports

The most quoted result is on LibriSpeech test-clean, a benchmark of read English audiobooks. There, the large-v2 Whisper model reaches a WER of 2.7%, exactly the same as a wav2vec 2.0 model trained specifically on LibriSpeech.

The interesting part is what happens outside that benchmark. The authors evaluated both models, without any fine-tuning, on a range of other English datasets: podcasts, meetings, phone calls, accented speech and more. Here is an extract of the paper's Table 2 (WER in %, lower is better):

Dataset wav2vec 2.0 Large (no LM) Whisper large-v2 Relative error reduction
LibriSpeech Clean2.72.70.0%
LibriSpeech Other6.25.216.1%
Common Voice29.99.069.9%
TED-LIUM10.54.061.9%
CHiME-665.825.561.2%
Switchboard28.313.851.2%
AMI SDM167.636.446.2%
Average (13 datasets, excluding LibriSpeech Clean)29.312.855.2%

The takeaway is not that Whisper is "better than humans" or "the most accurate system". It is that two models that look identical on a clean benchmark can behave very differently on messy audio. Averaged over the other datasets, Whisper made 55.2% fewer errors. The paper also notes that its smallest model, with only 39 million parameters, is roughly competitive with the best supervised LibriSpeech model once you leave LibriSpeech.

The authors also compared Whisper with commercial speech recognition services on long recordings, queried in September 2022, and found it performed better than the compared systems on most datasets. They add a fair caveat: some commercial systems may have been trained on those public datasets, so the comparison may not reflect true robustness. Those services have changed since, so treat that comparison as a snapshot.

Accuracy Across Languages

Whisper is often described as multilingual, and it is, but unevenly. The official repository states plainly that performance "varies widely depending on the language", and publishes per-language error rates for large-v3 and large-v2 on the Common Voice 15 and Fleurs datasets.

The paper explains why. Across languages, accuracy is strongly predicted by how much training audio existed for that language: the authors estimate that the error rate halves for every 16-fold increase in training data. Because the dataset came mostly from English-centric parts of the internet, most languages had less than 1,000 hours of training audio. Widely spoken languages with a lot of online content tend to do well; lower-resource languages do noticeably worse.

The later large-v3 model, announced in November 2023, was trained on 1 million hours of weakly labeled audio plus 4 million hours of pseudo-labeled audio. Its announcement reports a 10% to 20% reduction of errors compared with large-v2, for the languages measured on Common Voice 15 and Fleurs.

Noise and Real-World Audio

The paper tested robustness by adding white noise or "pub noise" (ambient sound and indistinct chatter, like a crowded restaurant) to LibriSpeech recordings. Many LibriSpeech-trained models beat Whisper when the audio was barely noisy, but they all degraded quickly and performed worse than Whisper once pub noise reached a signal-to-noise ratio below 10 dB.

In practice, this means a robust model is more forgiving of everyday conditions, but no model is immune. A closer microphone, such as a headset, still makes a bigger difference than any model choice when you dictate somewhere loud.

Model Sizes: Speed vs Accuracy

Whisper comes in several sizes. The official repository lists them as follows, with approximate memory needs and speed relative to the large model:

The turbo model is an optimized version of large-v3 with faster transcription and little loss in accuracy, but it is not trained for translation. Smaller models are easier to run on modest hardware, at the cost of accuracy, so any accuracy figure only makes sense next to the model size it was measured with.

Known Limitations

The paper is candid about what still goes wrong. Its limitations section lists errors that look "decidedly non-human":

Like any speech recognition system, it can also stumble on rare names, brand names and technical vocabulary. None of this cancels its strengths, but it explains why you should read a dictated text before sending it.

What This Means for Voice Typing on Android

For everyday dictation, the lessons from Whisper's research apply to any speech recognition engine: benchmark numbers on clean audio say little about your own voice, accent, language and room. The best test is to dictate a few real messages in the conditions you actually use.

Accuracy is also only part of the experience. Automatic punctuation, how easily you can correct a word, and whether dictation is available where you type often matter just as much. Our guide to voice dictation apps for Android compares the options, and voice typing vs keyboard looks at when speaking beats typing.

DictoKey is one option: an Android voice keyboard and floating dictation bubble that uses cloud-based AI speech recognition (so it needs an internet connection), restores punctuation automatically, and can rewrite or translate what you dictate in 51 languages. It is built first for people for whom typing is painful, slow or unreliable; see our accessibility page.

Try Voice Typing on Your Android Phone

DictoKey: voice keyboard plus floating bubble, automatic punctuation, rewriting and translation in 51 languages, no account.

Download on Google Play Free plan with a daily dictation limit — Android 10+

Frequently Asked Questions

What is Whisper?+
Whisper is an open-source speech recognition model released by OpenAI in 2022. It is a Transformer sequence-to-sequence model trained on 680,000 hours of audio collected from the internet. Its code and model weights are released under the MIT License, so anyone can download and run it.
How accurate is Whisper?+
In the original paper, the large-v2 model reaches a WER of 2.7% on LibriSpeech test-clean, the same as a strong LibriSpeech-trained model. The bigger difference shows up on other datasets: across 13 other English test sets, Whisper made 55.2% fewer errors on average than that model. Accuracy still depends heavily on the language, the audio quality and the model size.
Is Whisper equally accurate in every language?+
No. The model card says performance varies widely by language. The paper finds that accuracy is strongly linked to how much training audio exists for a language, and that most languages had less than 1,000 hours of training data, because the dataset was English-heavy.
What mistakes does Whisper make?+
The paper lists repeat loops, skipping the first or last few words of an audio segment, and occasional hallucination, where the model outputs text unrelated to the audio. These errors show up particularly in long recordings. Like other speech recognition systems, it can also misspell rare names and technical terms.
Which Whisper model size should I use?+
It is a trade-off between speed and accuracy. The official repository lists six sizes, from tiny (39 million parameters, about 1 GB of video memory) to large (1,550 million parameters, about 10 GB). The turbo model is an optimized version of large-v3 that is much faster with little loss in accuracy, but it is not trained for translation.
Does voice typing on Android use Whisper?+
It depends on the app. Some keyboards and dictation apps build on Whisper, others use different speech recognition engines. DictoKey, for example, uses cloud-based AI speech recognition and needs an internet connection. Check each app's own documentation rather than assuming which model it runs.

Sources

Checked on 2026-10-19.

Whisper and OpenAI are trademarks of their respective owner. DictoKey is an independent app and is not affiliated with or endorsed by OpenAI.