What it does
WhisperX is an open-source speech recognition tool built around Whisper. The README points to two gaps in plain Whisper: timestamps sit at the utterance level and can be off by seconds, and batching is not supported natively. WhisperX uses a faster-whisper backend with batched inference and aligns the words with a wav2vec2 phoneme model to produce word-level timestamps. It can also label speakers using pyannote-audio diarization, which needs a Hugging Face access token and acceptance of the model's user agreement, and it applies voice activity detection beforehand to reduce hallucination. You run it from the command line or from Python, where transcription, alignment and speaker assignment are separate steps. Alignment models are language-specific, with defaults for English, French, German, Spanish and Italian and many more languages through Hugging Face. Install it with pip. GPU use needs the CUDA toolkit 12.8, and a CPU mode is available, including on Mac OS X. The README lists limitations: words containing digits or currency symbols cannot be aligned, overlapping speech is handled poorly, and diarization is far from perfect.
Capabilities
- Word-level timestamps through wav2vec2 alignment
- Speaker labels from pyannote-audio diarization
- Batched inference on a faster-whisper backend, reported at 70x realtime with large-v2
- Voice activity detection before transcription
- Command-line and Python interfaces
- Runs on GPU or CPU
When to use it
- You need precise word timings for subtitles or highlighted transcripts
- You need to tell speakers apart in a recording
- You transcribe long audio and want faster batched processing
