microsoft/vibevoice

VibeVoice is a family of open-source AI models from Microsoft for voice-related tasks, covering both speech-to-text (converting spoken audio to written transcripts) and text-to-speech (converting written text into natural-sounding spoken audio). These are the same fundamental tasks used in products like transcription services, voice assistants, and audiobook generators, but released as open model weights that anyone can download and run or fine-tune. The project contains two main components. The ASR (Automatic Speech Recognition) model, called VibeVoice-ASR, is designed to transcribe up to 60 minutes of continuous audio in a single pass rather than breaking it into short chunks like most transcription tools. It outputs structured transcripts that include not just what was said but who said it (speaker identification, also called diarization) and when (timestamps), with support for custom vocabulary terms like technical jargon or proper names. The TTS (Text-to-Speech) model was capable of generating up to 90 minutes of speech with multiple distinct speakers in a single pass, but the TTS model code was removed from the repository in 2025 after it was found to be misused in ways inconsistent with responsible AI principles, a streaming real-time variant remains available. Both components use a novel architecture involving continuous speech tokenizers operating at a low frame rate to handle long audio efficiently. You would use VibeVoice ASR when you need to transcribe long interviews, meetings, podcasts, or lectures with speaker attribution, especially if you want fine-grained control over the model via fine-tuning on domain-specific audio. It is available via the Hugging Face Transformers library and integrates with vLLM for fast inference. The code is written in Python and the model weights are hosted on Hugging Face.