babysor/mockingbird

MockingBird is a Python-based AI voice cloning tool that can clone a person's voice from a short audio sample and then generate new speech in that cloned voice from any text you provide, in real time. The problem it solves is that training a voice synthesis model from scratch for a specific person's voice requires large amounts of data and time, MockingBird reduces that to just a few seconds of audio input. The system is built on a three-stage architecture common in modern text-to-speech research. First, an encoder model converts a short voice sample into a numerical representation of that speaker's unique vocal characteristics. Second, a synthesizer model (which the project specifically trained on Chinese Mandarin datasets including aidatatang_200zh, magicdata, and aishell3) takes text and the speaker representation and produces mel spectrograms, a visual representation of sound frequencies over time. Third, a vocoder model converts those spectrograms into actual audio waveforms. The pre-trained encoder and vocoder can be reused directly, only the synthesizer needs to be swapped for a Chinese-compatible version. A graphical toolbox and a web server interface are both available for running inference. The README notes the repository is no longer actively maintained, and the author has moved this work to a commercial service at noiz.ai. You would use this repository if you want to experiment with real-time Chinese Mandarin voice cloning locally, or if you want to study the architecture of a complete voice synthesis pipeline. The tech stack is Python, using PyTorch as the deep learning framework. A GPU is recommended for reasonable performance, though CPU operation is possible. Windows, Linux, and macOS (including Apple Silicon via Rosetta) are supported.