Robust Speech Recognition via Large-Scale Weak Supervision

Why This Matters

Speech recognition has made massive leaps using self-supervised pre-training (e.g., Wav2Vec 2.0). However, because these models are trained purely on raw audio, they lack a unified decoder capable of mapping features to text. This forces practitioners to fine-tune them on small, highly curated datasets like LibriSpeech.

The catch? Fine-tuning makes models highly fragile. They learn to exploit dataset-specific quirks (e.g., spelling rules, background noise levels) and achieve "superhuman" accuracy on held-out test sets, but fail spectacularly when deployed in the real world. Whisper shifts the paradigm: instead of unsupervised pre-training followed by supervised fine-tuning, why not train on massive, diverse internet audio with weak supervision?

The Big Idea

Whisper's core hypothesis is that sheer dataset size and diversity can override the noise of weak labels. By curating 680,000 hours of audio transcripts from the web, the model encounters thousands of recording environments, accents, and languages. To handle this multi-domain data without building specialized systems for Voice Activity Detection (VAD), Language Identification (LID), or Translation, Whisper frames all tasks as a single sequence-to-sequence task.

How It Works

1. Data Pipeline & Minimalist Preprocessing

Unlike traditional pipelines that normalize text (e.g., converting "68 million" to "sixty-eight million"), Whisper trains on raw text transcripts. This lets the model directly output natural punctuation and capitalization.

However, web data is noisy. The authors designed automated heuristics to filter out bad labels:

• Removing machine-generated transcripts (detected by lack of punctuation/capitalization).
• Using language detectors to ensure audio and transcript languages match.
• De-duplicating transcripts to prevent overfitting.

2. Model Architecture

Whisper uses a classic encoder-decoder Transformer. The audio processing pipeline is standard:

$$x_{\text{mel}} = \text{Log-Mel}(x_{\text{audio}})$$

The input audio is converted to an 80-channel log-magnitude Mel spectrogram computed on 25ms windows with a 10ms stride. The encoder uses a small stem consisting of two convolutional layers with a filter width of 3, followed by sinusoidal positional embeddings.

Results & Benchmarks

The headline result of Whisper is not its state-of-the-art performance on clean data, but its generalization robustness. While standard supervised models match Whisper on LibriSpeech clean-test (~2.7% WER), they degrade rapidly on other datasets.

Whisper achieves an average of 55.2% relative error reduction compared to supervised models when evaluated out-of-distribution, closing the gap to human performance.

Limitations & Open Questions

• Hallucinations & Repetition Loops: Because the decoder is an autoregressive language model, it can get stuck in repetition loops or generate plausible but completely fabricated text when audio is silent or highly noisy.
• Low-Resource Languages: Performance remains poor for languages with less than 1,000 hours of training data in the corpus.
• Long-Form Chunking: Whisper processes audio in 30-second chunks. Transcribing long-form audio requires complex buffering and temperature-scheduling heuristics to avoid alignment drift.

Glossary

Citation

@article{radford2022robust,
  title={Robust speech recognition via large-scale weak supervision},
  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  journal={arXiv preprint arXiv:2212.04356},
  year={2022}
}