mormolykos/ttsproof

TTSProof is a tool that checks whether text to speech (TTS) audio actually sounds right, instead of just measuring word accuracy. A normal accuracy score can miss real problems, like a clip that is empty, cut off short, clipped, stuck repeating itself, or far too long. At the same time, a plain word error score can flag audio as wrong when it isn't, just because the written text said 3:30 PM and the transcript said three thirty pee em. The tool works in three layers. First, it runs structural checks on the audio file itself: is it empty, too short, has long silences, is clipped, or loops on a repeated chunk. These checks use only numpy and soundfile, no AI model required. Second, it compares the expected text against what a speech recognizer heard, but first converts both into the same spoken form, covering numbers, dates, times, and acronyms, so formatting differences do not count as mistakes. Third, when the audio is clean but the speech recognizer disagrees on something very short, like a single letter or an acronym, the sample gets set aside for a human to check rather than automatically marked as a failure. TTSProof ships with a built in benchmark corpus of 817 test cases across categories like numbers, dates, phone numbers, names, tongue twisters, and more, so a user can score any TTS engine with a single command. It prints a scoreboard by category in the terminal and generates a report page with audio players for any failures. It can also compare results across different TTS engines and gate a continuous integration pipeline by failing the build when quality drops compared to a saved baseline. The method behind TTSProof was tested on a real production TTS service across 390 audio samples and published as a citable technical report with a DOI. It is installed with pip, either with or without the optional speech recognition support. The project's license is MIT, so it can be used freely, including for commercial purposes.