What it does
Whisper is an automatic speech recognition system that OpenAI introduced in September 2022. It was trained on 680,000 hours of multilingual, multitask supervised audio collected from the web, which the page credits for better robustness to accents, background noise and technical language. The model is an encoder-decoder Transformer. Audio is split into 30-second chunks, converted to a log-Mel spectrogram and passed to the encoder, and the decoder predicts text mixed with special tokens that select tasks such as language identification, phrase-level timestamps, multilingual transcription and translation into English. OpenAI says it is open-sourcing the models and inference code as a base for applications and further research. It notes that Whisper does not beat models specialized for the LibriSpeech benchmark, but in zero-shot tests across many diverse datasets it makes about half as many errors as those models. About a third of the training audio is non-English.
Capabilities
- Transcription in multiple languages
- Translation of speech into English
- Language identification and phrase-level timestamps
- Trained on 680,000 hours of web audio
- Open-sourced models and inference code
When to use it
- Adding voice interfaces to applications
- Transcribing audio with accents, background noise or technical vocabulary
- Translating non-English speech into English text
- Using a robust speech model as a base for further research
