m-bain/whisperx

WhisperX is an open-source tool that turns recorded speech into text and tags each word with a precise timestamp, so a transcript lines up tightly with the audio. It builds on Whisper, a speech-recognition model from OpenAI that is good at the words themselves but only gives rough timing for whole utterances and cannot natively process audio in batches. WhisperX adds the missing pieces: accurate per-word timing, fast batched inference, and the ability to tell different speakers apart in a conversation. The README describes a small pipeline. First, Voice Activity Detection finds the chunks of audio that contain speech and skips silence, which speeds things up and cuts down on the model "hallucinating" words that were never said. Those chunks are then transcribed in batches using a faster-whisper backend, which the project says reaches around 70x realtime with the large-v2 model. Next, a phoneme-based recognition model (a sibling family of models trained on the smallest sound units in language, such as the p in "tap") is used for forced alignment: lining up the transcript against the audio so each word gets an exact start and end time. An optional speaker-diarization step from pyannote-audio then splits the audio by speaker and attaches speaker labels to the transcript. You would use WhisperX when the timestamps matter as much as the text: making subtitle files for video, building searchable archives of podcasts or meetings, or feeding precisely timed transcripts into downstream tools. It is a Python package installable from PyPI, runs on GPU through CUDA or on CPU, and uses a Hugging Face token to enable diarization.