soul-ailab/soulx-transcriber

SoulX-Transcriber is an AI research release from Soul AI Lab and Northwestern Polytechnical University that tackles the problem of transcribing conversations involving multiple speakers. The challenge in multi-speaker transcription is producing output that answers three questions together: who spoke, when did they speak, and exactly what did they say. Most existing systems handle these questions in separate steps, which can cause errors to compound. SoulX-Transcriber trains a single model to answer all three at once. The model is a large audio language model, meaning it processes audio directly rather than converting it to an intermediate representation first. It outputs structured transcripts that include timestamps, speaker labels, and text for each utterance. A key focus of the training approach is handling real-world conversation difficulties: overlapping speech (where two people talk at the same time), fast speaker turns, and confusion between speakers with similar voices. The project reports benchmark results on three publicly available multi-speaker datasets: AISHELL-4 and AliMeeting (which are Mandarin Chinese meeting recordings) and AMI-SDM (an English meeting dataset). On AISHELL-4 and AliMeeting, SoulX-Transcriber posts the best diarization error rates (a measure of how often the model assigns speech to the wrong speaker) among the systems compared, while also achieving competitive word error rates. Comparisons include Gemini models and Qwen audio models as baselines. The repository includes the code for running transcription using the released model weights, which are hosted on Hugging Face. The training approach involves two stages: a multi-task pre-training phase that builds speaker awareness, followed by supervised fine-tuning. The project also describes a pipeline for generating synthetic training data from existing audio by matching speaker characteristics to produce more natural simulated dialogues. The model is Apache 2.0 licensed. Researchers interested in reproducing the results can download the model weights from Hugging Face and run inference using the provided scripts. A demo page with audio examples is linked from the README. The full README is longer than what was shown.